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Executive Summary 


Technology is rapidly transforming how people learn and how we provide training. For 
example, in higher education, there are estimated to be nearly 6.7 million students enrolled in 
online education courses (National Center for Education Statistics Fast Facts, 2018). This 
substantial number of students accounts for 33.7% ofthe current student population (National 
Center for Education Statistics, 2018, Table 311.15) and are part of a two-decade long growth 
trend for online learning. Technology is changing even traditional classrooms. Flipped 
classrooms and technology-enhanced classrooms offload much of the direct instruction using 
distributed resources, with instructors taking on the role of “guide” or “facilitator.” This rapid 
growth and change in student populations has made educational technology into a $7.5 billion 
industry as of 2019. 


The dynamic and high-stakes nature of education and training software demands that we rely on 
evidence-based practices and policies to make decisions about technology adoption and 
implementation. The purpose of the Science of Learning and Readiness project (SoLakR) is to 
identify these practices. This State-of-the-Art Report (SoAR) is the first project deliverable and 
summarizes evidence-based practices and implementation within public, private, and academic 
sectors. The SoAR also includes specific guidance on metrics-based, most effective, at scale, and 
blended learning strategies within institutional systems, courseware, and pedagogical methods. 
Methodologically, this report emerges from a broadly scoped review process comprising over 
200 formal database searches. Our search strategy was inclusive of academic, military, and 
industry resources. 


Overarching findings demonstrate that (1) fundamental principles of human learning from the 
learning sciences are applicable to blended and learning-at-scale environments, (2) human 
learning within these environments must be supported by technology, (3) the technology must 
report data on the learning process to the learning organization, and (4) learning organizations 
must use data to (a) support learners with learning, social, and academic guidance, and (b) 
support members of the learning institutions with training, support, and recognition. 


To further understand perceived importance and implementation of these practices, we 
surveyed learning organizations representing public, private, and academic sectors. We 
observed a consistent discrepancy: respondents reported that actual implementation of best 
practices fell short (i.e., ratings of implementation were significantly lower than ratings of 
perceived importance). This pattern was most striking within military organizations. The survey 
sample was small and thus limited in generalizability. Nonetheless, the consistently high 
perceived importance of best practices in the public sector—particularly the military—suggests 
a readiness for a transition to advanced distributed learning methodologies. 


SoLaR-SoAR consists of a review of the current state of the art for distributed learning 
environments. The report is structured for use by multiple types of end users. The main report 
provides a 22-page, high-level overview of findings of the current state of the art. This section 
can serve as a quick reference. The report’s Appendixes provide a detailed summary of the 
empirical literature on state-of-the-art distributed learning ecosystems. These materials enable 
“deeper dives” into learning organization, technology, and human pedagogy topics. 


State of the Art Report: The Current State of 


Blended Learning and eLearning at Scale 
—SE EE 


Report Overview 


Learning organizations are rapidly changing how they enable learning and provide training. 
These changes are driven by both technological innovations and the need to provide education 
and training to larger numbers of learners at a rapid pace (Graesser, Hu, & Ritter, 2019). Many of 
these learners are immersed in online learning environments. For example, there are an 
estimated 6,651,536 students enrolled in online education courses at the postsecondary level in 
the United States (National Center for Education Statistics Fast Facts, 2018), and these 6.7 million 
students account for 33.7% of the current student population (National Center for Education 
Statistics, 2018, Table 311.15). These numbers are indicative of a growth trend of online learning 
that has continued for the last 13 years in the United States (Allen & Seaman, 2013). Moreover, 
even traditional classrooms are changing—increasingly use technology to offload direct 
instruction and allowing instructors to facilitate higher level learning (e.g., flipped classrooms 
and technology-enhanced classrooms) (Enfield, 2013; Roehl, Reddy, & Shannon, 2013). 


The high pressure of providing education and training within this rapidly growing technological 
environment often requires rapid decisions based on limited information. Unfortunately, such 
demands can result in well-meaning decision makers pursuing suboptimal or fallacious choices. 
Decision makers often cling to traditional methods (e.g., in-person lectures) instead of 
innovating (Allen & Seaman, 2013), in part due to beliefs that eLearning and flipped/technology- 
enhanced classrooms are less effective. This is not true. eLearning (Means, Toyama, Murphy, & 
Baki, 2013) and blended /flipped/technology enhanced classrooms (Liu, Peng, Zhang, Hu, Li, & 
Yan, 2016) can be just as effective as traditional classrooms and, in some cases, more effective. 
However, to be successful, there must be a deliberate consideration of the needs of learners and 
the organization, support for those needs, and willingness to explore state-of-the-art techniques 
for addressing the needs. 


This report presents a state-of-the-art exploration of distributed learning environments. For this 
report, we define a “State-of-the-Art Distributed Learning Environment” as a learning ecosystem 
that is supported by technology and educational theory/findings. To be considered state of the 
art, components within the ecosystem must 

have (a) empirical evidence of effectiveness “Three pillars [for scaling eLearning] include 
(i.e., data) and (b) evidence ofimplementation | content, operation, and technology. The three 


pillars are the key to have a real solution to 


: personalized learning. Moreover, business, 
Heataung ecosyeteny OW elcutt siechats, 202°); people, and data are the three foundations of 


learning organizations must focus on best deploying the personalized learning.” 
practices that span the organizational or Richard Tong, Chief Architect 


enterprise level, the technology level, and the Squirrel Al Learning 
human level. Key evidence-based and state-of- 
the-art practices are summarized in the main report; expanded reviews of relevant research are 


(i.e., application). To establish a modern 
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subsequently available in Appendix A (Institutional Support), Appendix B (Technology), 
Appendix C (Human Learning). 


Human learning has not changed, but technological support has 

While technology has become a more important component within the learning process, the 
fundamental principles of how humans learn have not changed in the last few decades. For 
humans, learning is messy. The act of teaching and learning does not take place in a sterile 
environment, nor can it take place automatically (Hattie, 2009). Learning is individualistic, 
sometimes spontaneous, but often very effortful, slow, and gradual, and moves forward in fits- 
and-starts (Hattie, 2009). Learning organizations must be established to support the needs of the 
stakeholders, ensure that appropriate resources are allocated, and that there must be buy in 
from all stakeholders (Giattino & Strafford, 2019; Muilenburg & Berge, 2001; Moore & Kearsley, 
2011). Thus, it is important for educational decision makers, instructional designers, and 
instructors to understand the best practices for learning and implement them to the best of their 
ability and resources. In the remainder of this section, we have summarized the basics of human 
learning that could be supported by well-organized, state-of-the-art distributed learning. For an 
expanded discussion on these issues, please see Appendix C: Distributed Online Pedagogy 
Review. 


“(We had to] turn Kaplan into a learning engineering organization that uses 
learning sciences and good evidence about learning in practical ways to 
iterate improvements for learning outcomes that were relevant to each 


different learning organization inside Kaplan.” 
Bror Saxberg, Vice President 
Learning Sciences at Chan Zuckerberg Initiative 


Guides for Human Learning 

This report highlights key areas of human learning that have been shown to impact learning with 
technology. There are numerous theoretical perspectives on human learning, such as 
behaviorism (rote association/practice), cognitivism (memory, encoding, and processing of 
information), and constructivism (build representation of knowledge) (Ertmer & Newby, 2013). 
Previous extensive summaries have offered actionable recommendations (see Alexander, 
Schallert, & Reynolds, 2009; Craig & Douglas, 2019; Graesser, 2009; Pashler et al., 2007), and 
expansive reviews can be found within How People Learn volume I: Brain, Mind, experience, and 
School (National Research Council, 2000) and How People Learn Volume 2: Learners, Contexts, 
and Cultures (National Academies of Sciences, Engineering, and medicine, 2018). The current 
report highlights and exemplifies aspects of human learning that are particularly salient to 
learning with technology. 


Human Cognitive Processing and Technology 

Principles of cognition have been applied to instructional design using many different 
approaches that are grounded in the understanding that basic human cognition consists of 
sensory memory, working memory, and long-term memory (Mayer, 2017), each with their own 
unique properties and limitations. Sensory memory functions as the receiver of stimuli, 
transmitting information to working memory where active manipulation and encoding take place 
(Mayer, 2009). Both sensory and working memory are limited in capacity and duration (Cowan, 
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2010; Mayer, 2017; Paas and Sweller, 2014). Information that is not actively attended to can be 
easily lost. In contrast, long-term memory is both expansive in scope and duration—this is where 
learned information is stored over time, whether days, weeks, years, or a lifetime (Paas & 
Sweller, 2014; Sweller, Ayres, & Kalyuga, 2011). However, the transition of information from 
working memory to long-term memory requires encoding. New information must be mentally 
organized and integrated with prior knowledge to be held in long-term memory (Mayer, 2017). 
Such encoding can proceed in multiple ways, including visual (e.g., images, scenes, and text), 
auditory (e.g., sounds), verbal (e.g., spoken or written words), semantic (e.g., conceptual and 
personal meaning), episodic (e.g., temporal sequences), and more. Indeed, memories are more 
robust when they are encoded in multiple ways or modalities, such as combining visual, 
auditory, and verbal memory traces simultaneously. 


Two prominent examples emerging from these concepts include cognitive load theory (CLT) 
and the cognitive theory of multimedia learning (CTML). In brief, CLT argues that the properties 
of instructional designs and activities impose burdens or “load” on human cognitive systems in 
different ways. One source of load is the inherent complexity of the material or learning task 
(i.e., intrinsic load), which is necessary and unavoidable. In contrast, the design of instructional 
materials may induce unnecessary burdens (i.e., extraneous load), such as distractions or clumsy 
interfaces, that require cognitive effort unrelated to learning. Finally, several models describe 
additional cognitive effort that is beneficial to learning (i.e., germane load). In some cases, 
learners might be asked to engage in tasks (e.g., self-explaining and self-questioning) that are 
more difficult than the core task (e.g., reading a text or listening to a lecture), but which engage 
them in deeper or more meaningful encoding. Importantly, regardless of the types of load, the 
essential argument of CLT is that learning is hindered when total cognitive load exceeds the 
working memory capacity of the learner (Paas & Sweller, 2014). Thus, designers must strive to 
balance necessary or beneficial task demands while minimizing wasteful or distracting. 


CTML expands and applies CLT principles to the design of multimedia learmmng materials, CTMIL 
emphasizes two processing pathways or modalities (Le, visual and auditory) that possess the 
own working memory capacity (Mayer, 2009). Each pathway can withstand a certam degree af 
“load” and can complement each other—disiributwng load across different modalities is better 
than overloading either system. Moreover, strategic processing and mtegrating information via 
both channels encourages multiple encoding and more robust recall and comprehension. Like 
CLT, research CTML is supported hy an expansive body of literature and generalizable 
instructional design principles (discussed in this report, also see Mayer, 2009, 2017). Indeed, 
multimedia learning has been an influential area of research over the last 30 years. These 
methods have applied in digital and computer-based settings and are easily transferable to 
eLearning environments (Mayer, 2017). They have also heen shown to be one of the most 
consistently effective technologies for learning at scale with 647 of reported results hemg 
positive Davis, Chen, Hauff & Houben, 2018). in sum, many research-hased instructional design 
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Appendix C: Distributed Online Pedagogy Review). 


Scaffolding and Instructional Guidance 

Learners are (almost by definition) individuals who lack robust knowledge or skill within a given 
domain. The goal is to help learners acquire these competencies and proficiencies. An important 
consideration, however, is how much support and guidance are provided to the learners. Should 
learners receive direct and structured instruction, or should they be encouraged to explore and 

discover within more open environments? 


Problem-based learning (PBL) and inquiry-based learning (IL) are two example paradigms in 
education where this debate has taken place. In brief, PBL presents learners with open-ended, 
complex, and real-world problems that must solve by researching the problem, acquiring 
necessary knowledge and skills, and then applying these resources to obtain a solution. PBL 
emerged from and is extensively used in medical education (e.g., Colliver, 2000; Schmidt, 2010). 
Similarly, IL presents learners with meaningful scientific phenomena and questions, and asks 
them to explore the domain to develop hypotheses, test their hypotheses, and gain necessary 
knowledge and skills to do so (Hmelo-Silver et al., 2007). Importantly, both PBL and IL can be 
implemented with very little direct guidance (i.e., primarily exploration) or with higher degree 
of structure and feedback at each stage. 


Hatti’s (2009) review of four meta-analyses and more than 200 studies found that inquiry-based 
learning produced an average effect size of d = 0.35 or an 14% increase over the average 
performance of controls. Importantly, implementation exhibits a significant influence on efficacy, 
which interacts with learners’ prior knowledge and skills. When learners’ knowledge is limited 
or they are provided no to little guidance, PBL and IL teaching methods are often ineffective 
(Kirchner, Sweller, & Clark, 2006). PBL and IL activities must be properly scaffolded to provide 
structure and support for the learners (Hmelo-Silver et al., 2007), which enables them to dissect 
complex problem cases into more reasonable pieces within the students’ zones of proximal 
development (Fernandez, Wegerif, Mercer, Rojas-Drummond, 2001; Hmelo-Silver et al., 2007). 
In contrast, unsupported “floundering” does not facilitate learning. 


Scaffolding in PBL and IL environments (generalizable to other learning settings) can take many 
forms, such as providing missing information or prompting students to reflect (Kim & Lim, 2019). 
Feedback is also essential for promoting growth and productive change (Alharbi, 2017). Hattie 
(2009) reported ranked feedback in the top ten factors that influence human performance out of 
100 surveyed. Notably, not all feedback is equally effective. Effective feedback must be 
consistent, specific, performance-focused, timely, purposeful, task-appropriate, and applied to 
future learning (Coll, Rochera, Gispert, & Diaz-Barriga, 2013; Harvey, Radomski, & O’Connor, 
2013; Shute, 2008). Finally, although teacher-to-student feedback is often the focus, Hattie (2009, 
2012) noted that student-to-teacher feedback was invaluable for helping teachers to adapt and 
improve their instruction. 


Motivation and Emotion 


One limitation of “cognitive” approaches to learning is that they sometimes (over)simplify by 


neglecting human motivational states or emotional responses, or by assuming that such states 
are consistent across learners. However, academic emotions (e.g., anxiety, confusion, boredom, 
and frustration) have been found to significantly influence cognition, learning, and learning- 


related processes (Craig, Graesser, Sullins, & Gholson, 2004; Pekrun, Goetz, Titz, & Perry, 2002; 
D’Mello, & Graeser, 2012). 


Five contemporary motivational theories have been particularly influential: expectancy-value 
theory, social cognitive theory, attribution theory, self-determination theory, and achievement goal 
theory. In brief, such theories articulate how human expectations, goals, psychological needs, 
comparisons to others, explanations of behaviors (both others’ and our own) shape our 
behaviors in complex ways. Learning behaviors and environments are no exception. For 
example, research on achievement goals describes how goals focused on seeking mastery (i.e., 
gaining skills and proficiency) and positive performance (i.e., gaining good grades and 
rankings) inspire better strategic effort and learning outcomes, whereas goals focused on 
avoiding all mistakes (i.e., perfectionism) or failure are associated with less effort and learning. 


There is evidence that motivational variables—as defined by the above motivational theories— 
can be important and predictive within learning analytics (Aguilar, 2016), MOOCs (Beaven, 
Hauck, Comas-Quinn, Lewis, & de los Arcos, 2014; Loizzo, Ertmer, Watson, & Watson, 2017; 
Martin, Kelly, & Terry, 2018), synchronous and asynchronous instruction (Lin & Overbaugh, 
2009), collaborative online learning (Du, Fan Xu, Wang, Sun, & Liu, 2019), vicarious online 
learning (Craig, 2018; Twyford & Craig, 2017; Craig, Sullins, Witherspoon, & Gholson, 2006; 
McNamara, Levinstein, & Boonthum, 2004), and other online learning environments (Chen & 
Jang, 2010; Kennan, Bigatel, Stockdale & Hoewe, 2018; Wang & Wu, 2008). Accordingly, there 
may be value in designing courses to foster learners’ motivation. Understanding theories of 
motivation, in conjunction with an understanding of the learners being taught, could help create 
effective distributed learning environments. 


The literature on motivation and learning is extensive, detailed, and nuanced. Although much of 
this research emerged from traditional “offline” and face-to-face courses, one might perhaps 
extrapolate to distributed learning contexts. However, although designing for emotions related 
to learning is an emerging area of study, there is not yet sufficient evidence to define concrete 
instructional design principles regarding emotions and learning (Mayer & Estrella, 2014). 


Self-Regulated Learning 

Self-regulated learning (SRL) refers to learners’ (primarily) self-directed efforts to organize, 
manage, and motivate their own learning processes and outcomes. Numerous theoretical 
perspectives and models for SRL have been articulated (Panadero, 2017), but they generally 
comprise similar sets of metacognitive and strategic activities (e.g., planning and analyzing 
tasks, performing tasks and enacting strategies, monitoring performance and learning, and 
adapting future learning efforts). SRL has been consistently linked to more successful and robust 
learning outcomes (Zimmerman & Schunk, 2011), although students frequently need external 
support to initiate or continue through the SRL process (e.g., feedback, Winne, 2005). 


In the domain of online higher education, Broadbent and Poon (2015) conducted a systematic 
review to examine the role of SRL strategies in academic achievement. These researchers 
identified only 12 studies examining self-regulation strategies: metacognition, time 
management, effort regulation, peer learning, elaboration, rehearsal, organization, critical 
thinking, and help seeking. Importantly, only four of these strategies were significantly 


associated with improved academic performance: metacognition, time management, effort 
regulation, and critical thinking. 


Learning analytics approaches are increasingly being used to automatically detect and/or 
promote SRL (Milligan & Griffin, 2016; Pardo, Han, & Ellis, 2017; Winne, 2018). For instance, 
learners’ SRL strategies (e.g., goal-setting and strategic planning) have been found to be 
predictive of behavior and goal attainment in massive open online courses or MOOCs (Kizilcec, 
Perez-Sanagustin, & Maldonado, 2017). However, despite positive initial results, research on 
self-regulation in learning at scale (e.g., in MOOCs) remains sparse (Wong, Baars, Davis, Van 
Der Zee, Houben, & Paas, 2019). 


More research is needed to understand the importance of SRL in distributed learning contexts. 
At this point, we cannot recommend any specific or generalizable interventions. Nonetheless, 
several approaches may be plausible or fruitful. For example, one approach might be to display 
learning analytics to learners via dashboards—learners’ SRL may be facilitated by having access 
to detailed information about their own learning behaviors, performance over time, or affective 
states. Revealing this information to learners might “offload” some of the challenges of self- 
monitoring, which might in turn facilitate self-regulation. Regulation might be further supported 
by personalized suggestions for adaptive behaviors (e.g., if procrastination is detected, then 
time-management techniques could be offered). 


Learning Platforms 


For this report, we examined state-of-the-art distributed learning within two categories of 
technology-enhanced learning platforms that support implementation of learning at scale: 
blended learning environments and online learning environments. Blended learning 
environments are face-to-face learning environments that provide part of the instruction using 
technology. Online learning is defined as learning within an online medium (e.g., internet or 
localized intranet). Learning at scale refers to serving large number of students (e.g., hundreds 
or thousands of students) within the same courses and/or at the same time. Our definition of “at 
scale” is somewhat broader than other definitions. For example, Roll, Russell, and GaSevi¢ 
(2018) define learning at scale as “the study of the technologies, pedagogies, analyses, and 
theories of learning and teaching that take place with a large number of learners and a high ratio 
of learners to facilitators.”” Our definition is more applied and practical instead of research 
based. Such environments can be viewed as a continuum that vary in the degree of human 
versus technological support. Blended learning environments tend to entail more human 
support, whereas technological support increases as learning environments move online. 


Levels of Online Learning 


Online learning can be defined as learning over the Internet or a digitally networked system. 
Importantly, online learning continues to provide students with direct connection to course 
instructors and other students, along with direct communication between them. Such 
communication differentiates modern online learning environments from older distance learning 
environments, wherein learners might install an isolated software program (e.g., an intelligent 
tutoring system) that never connected to a larger network (see Graesser, Hu, & Ritter, 2018 for 
detailed history). In addition, once fully online delivery has been attained, online learning 


environments can rapidly offer instructional resources to large numbers of students via (a) 
system infrastructures that support many users and (b) deploying additional instances of the 
course. The primary limitation involves the amount of human instructor contact needed to 
support the learners. 


Smith et al. (2007) suggested a four-level taxonomy for classifying online learning based on the 
percent of materials that are online: (1) web-enhanced courses or environments that use minimal 
web elements (e.g., LMS syllabus or announcement features); (2) blended courses that provide 
online documents yet hosts less than 45% of course activities online; (3) hybrid courses that 
deliver 45% to 80% of class activities online; and (4) fully online courses or environments in 
which more than 80% of activities and content are online. Due to their higher levels of both in- 
person and online interaction, this report considers Levels 1 through 3 to be forms of ‘blended 
learning.” The in-person human resources needed to support such instruction induces 
constraints on scale-up. In contrast, this report considers the fourth level to be “true” online 
learning with the best potential for implementation at scale. 


Levels of Technology Integration 


In addition to the percentage of course materials 
available and supported online, another essential “If you look at the current technology, for 
factor is the nature of technology integration. example learning management systems, a lot 


es ; : . of the current digital learning capabilities are 
Specincally, me neve Which teenie y eed 2 still built on top of an instructor-centric model. 
support teaching and learning. 


However, we need to understand the individual 


The Substitution Augmentation Modification Beeo cad proee suppor acorn et pol 
a granular level where customized learning 


Redefinition Model (SAMR; Romrell, Kidder, & experiences are for everyone and a global 


Wood, 2014) provides a useful framework for level where customized whole curriculum is for 
evaluating such usage. each one of them.” 

Richard Tong, Chief Architect 
Substitution describes the most common approach, Squirrel Al Learning 


wherein technology replaces existing resources or 

tasks. For example, instructors might use PowerPoint and projectors to replace acetate slides 
and chalkboards, or students replace paper notebooks with laptops. Instructors and 
organizations often use this substitution method to facilitate fast transition from face to face to 
online instruction. At this level, instructors take class materials and place them online for 
students to access. 


The second level is augmentation, in which technology resources not only replace existing 
resources but also offer improvements. For example, instead of merely sharing online eBooks or 
lecture notes, instructors might also provide multimedia videos. Such videos allow students to 
view information with greater control (e.g., pausing, rewinding, and fast-forwarding), and can 
also afford communication of dynamic processes and demonstrations (unlike static text or 


presentations). Many traditional eLearning systems do not go beyond this level. 


In modification, technology is used to transform or modify the learning process. One approach is 
to use technology to reinstate elements of face-to-face learning that are lost in the online 
transition. For example, a virtual collaborative workspace might be provided that allows 
students to engage in collaborative discussion and co-construction of ideas. 


The highest level is redefinition, in which technology enables new learning processes or tasks 
that were previously impossible or inconceivable within face-to face learning settings. For 
instance, technology might offer just-in-time feedback, learner modeling and tracking, data 
visualizations, or personalization. The redefinition level is key characteristic for state-of-the-art 
distributed online learning systems—such systems are either redefining learning via technology 
or attempting to move in this direction. 


In sum, most traditional eLearning can be categorized at the substitution or augmentation levels. 
However, state-of-the-art practices for improving blended learning and online learning, and 
scaling up distributed learning environments, involve intentional design for modification and 
redefinition levels. 


The SAMR model highlights our tendency to 
use new technologies in old-school ways. 


REDEFINITION 


Technology enables new tasks, previously inconceivable 


MODIFICATION 


Technology enables significant task redesign 


AUGMENTATION 


Technology acts as direct substitute, with functional improvement 


SUBSTITUTION 


Technology acts as direct substitute, with no functional improvement 


Figure 1. Open source image summarizing SAMR model reused from Craig & Douglas, (2019). 
Distributed learning instructional theories. In Walcutt, J.J. & Schatz, 8. (Eds.). Modernizing 
Learning: Building the Future Learning Ecosystem (pp. 43-60). Washington, DC: Government 
Publishing Office. Creative Commons Attribution CC BY 4.0 IGO. 


Scale Up of Online Learning: From Traditional eLearning to Learning at Scale 


From large scale lecture halls that can hold hundreds of participants to massive online open 


courses (MOOCs), learning at scale is not a new issue. Numerous interventions have been 
explored as educators sought to effectively, efficiently, and simultaneously educate or train 
large numbers of learners. This report focuses on the state of the art for distributed learning 
environments. Consequently, we focus our discussion on Jearning at scale on MOOCs because 


these settings have (a) very broad implementation (i.e., currently over 4000 active users), a 
rapidly growing research base, and (c) clear potential for learning at scale (Davis et al., 2018). 


In brief, MOOCs are online courses that can host very high enrollments. These courses were 
originally thought to be a revolution in education by providing free, accessible information to 
everyone with an Internet connection. However, MOOCs have faced a variety of challenges, 
including low completion rates and limited retention after the first year (Reich & Ruipérez- 
Valiente, 2019). These issues have caused researchers to question what success looks like in 
MOOCs (Aparicio et al., 2019). In addition, it is important to understand why learners enroll in 
MOOCs in the first place (i.e., learning goals). Do students intend to complete or master the 
entire course, or are they seeking only specific skills or pieces of knowledge (and thus 
disregard the remaining content)? 


Our review concluded that MOOCs are not the answer for every distributed learning ecosystem. 
Even with advanced learning analytics and capacity for personalization (e.g., xAPI), MOOCs are 
not appropriate for every context. Although any content delivered fully online might potentially 
be scalable, it is important to consider what content should be scaled and how scale up should 
be achieved. 


Researchers have proposed using Bloom’s classic taxonomy of learning objectives as a 
framework to answer these questions (Taft, Perkowki, & Martin, 2011). Bloom’s taxonomy 
comprises six basic levels: knowledge, comprehension, application, analysis, synthesis, and 
evaluation (Bloom, Englehart, Furst, Hill, & Krathwohl, 1956). The knowledge level includes basic 
recognition and recall of that information, whereas comprehension entails the ability to 
demonstrate understanding of information. At higher levels, application involves using 
information to solve problems, analysis requires understanding the underlying components and 


relationships between information, and synthesis involves integrating and organizing diverse 


elements of information. At the highest level, evaluation involves forming arguments based on 
the information and/or evaluating work using the information. Taft et al. (2011) recommended 
that information on the lower levels of Bloom’s scale (i.e., knowledge and comprehension) is 
more readily taught and learned in larger class sizes (e.g., MOOCs or other at scale classrooms). 
In contrast, it perhaps inappropriate or difficult to scale up courses that require students and 
faculty to work at higher levels of the taxonomy (i.e., analysis, synthesis, and evaluation) or that 
require constructive and interactive learning (Chi & Wylie, 2014). Online classes that teach 
complex content can reach larger numbers of students by offering multiple sections, but this 
approach also demands significantly more instructors and instructor time. 


In sum, considerations of human resources, technology resources, and levels of information 
complexity suggest that learning at scale may be constrained by content area or population—not 
every topic or course is readily scalable. Nonetheless, across traditional eLearning, blended 
learning, and online learning at scale approaches, general recommendations might be broadly 
applicable to many distributed learning ecosystems. The following sections consider several 
such themes related to issues of institutional support, courseware, and distributed learning 


pedagogy. 


What is State of the Art for Scaling Up eLearning? 


Inspired by the SAMR model, modification and redefinition levels of technology integration are 
critical for scaling up traditional eLearning. This section summarizes the evidence-based and 
state-of-the-art practices for learning at scale revealed by our review of literature. This section is 
organized into current practices in three categories: institutional, technological, and pedagogical. 


Institutional Practices 


“To turn vision into executable deployment and the whole solution, a lot of 
the policy and procedures need to be focused on how to achieve that vision 
and reduce common risks or obstacles that are preventing [the vision] from 
happening.” 
Richard Tong, Chief Architect 
Squirrel AI Learning 


Learning Expertise within the Institution 

Although new technologies enable new tasks, interactions, and ways of thinking, the 
fundamental mechanisms underlying human learning remain unchanged. Technology tends to 
change the learning ecosystem by offloading unproductive tasks and focusing attention on tasks 
that afford active and constructive learning (Craig & Douglas, 2019). Consequently, effective 
technology implementation ideally requires that all levels of an organization (i.e., including, 
administrators, subject matter experts, instructors, etc.) possess a basic understanding and 
commitment to human learning principles. This culture must be established and reinforced by 
top-level administrators and then supported throughout the organization (Erb & Shah, 2019). It is 
also useful for all decision-makers to possess an understanding of good practices. 


Of course, it is not realistic for all individuals within a learning organization to be “learning 
experts,” and is unlikely that top-level administrators will be well-versed in learning theory and 
practice (Dooley & Murphrey, 2000). An essential compromise is that learning institutions must 
include at least a few individuals with detailed knowledge of learning principles along with the 
trust and authority to support implementation within the institutional network (Sohoni & Craig, 
2016). This role should not (perhaps cannot) be filled by a single person. Optimally, such 
expertise should be provided by group of individuals or teams distributed throughout the levels 
of the organization (Kurzweil & Marcellas, 2019). Example roles include educational specialists 
serving as higher-level directors, learning engineers, instructional designers, or 
SMEs/instructors who are domain-based educational researchers (DBER). These individuals 
should not represent an isolated unit but should participate in an integrated and interconnected 
network that supports the overall learning organization (Sohoni, Craig, & Vedula, 2017). 


Establish Trust 

In addition to promoting a basic understanding of human learning, organizations must also foster 
a sense of trust at all levels. In this case, “trust” broadly encompasses confidence in and positive 
appraisals of available technologies, as well as confidence and positive attitudes between 
organization members. 


Administrators can encourage trust, foster relationships, and seek common ground for 
discussion and action between stakeholders, while also collecting and using data to facilitate 
change and support faculty in the online education endeavor (Burnette, 2015). To be 
“trustworthy,” administrative decision making should be guided by evidence-based tools and 
metrics, such as the UPCEA Hallmarks of Excellence in Online Leadership, Quality Matters Program 
Rubrics, and the Online Learning Consortium Scorecard for the Administration of Online Programs 
(Cook & Uranis, 2019). Another foundation for trust is reciprocity (Levine, 2003). Evaluation of 
stakeholder performance should be fair and transparent, and evaluations should incorporate 
feedback from stakeholders (Berk, 2013). The overall process should be grounded in policy 
(Giattino & Stafford, 2019; Hai-Jew, 2006) that includes recognition of stakeholders’ contributions 
(e.g., compensation or acknowledgement of time commitment) (Muilenburg & Berge, 2001; 
Roby, Ashe, Singh, & Clark, 2013). 


Trust is also crucial at the student level. Students’ trust in instructors (Cavanagh, Chen, Bathgate, 
Frederick, Hanauer, & Graham, 2018) and perceived relevance of class content (Hai-Jew, 2007) 
have been shown to directly impact course grades. Hai-Jew (2007) suggests several methods for 
developing and maintaining student trust, including social engineering of the learning 
environment (i.e., building logical class structures that minimize negative events), frequent 
communication, maintaining a positive and consistent instructor persona, supporting peer-to- 
peer mutual dependence (e.g., collaboration), involving students in decision-making and 
communication, defining clear policies, and creating clear and transparent oversight. 


Human-centered Fvaluation 

Modern learning ecosystems are large, complicated structures with diverse stakeholders. To 
serve the entire organization and make informed decisions, it is essential to understand the 
needs of distinct groups (Dooley & Murphy, 2000) and how those groups are impacted by 
elements of the learning organization (Giattino & Stafford, 2019). In other words, evaluations 
(e.g., of learning, feasibility, technology adoption, and productivity) must consider the “human 
side” of the environment. 


Human-centered evaluations have been used to evaluate the functionality of computer and 
technological systems (e.g., usability and human-computer interaction, Nielson & Molich, 1990; 
Norman, 2013; Roscoe, Cooke, Branaghan, & Craig, 2017), and the same techniques can be 
implemented to collect data on how humans function within a learning ecosystem (Roscoe et al., 
2019). Human-centered evaluations within an eLearning courses might take various forms. 
Usability evaluations can enroll students in online class shells, and record errors and navigation 
behaviors as they locate materials and perform tasks. Such usability tests can span observational 
methods (e.g., digital observation via screen capture software) or think-aloud procedures (e.g., 
via videoconferencing) wherein students talk about what they are attempting to accomplish. 


Within a larger organization, survey methods can be used to develop an understanding of 


general knowledge or perceptions about proposed implementations. Many of these techniques 
and their uses are described in the UI/UX considerations subsection of Appendix B: Courseware 
& Distributed Technology Review. 


Flexible Class Sizes 


Although learning at scale aims to provide worthwhile instruction to larger numbers of learners, 
this goal does not mean that class sizes can grow infinitely. Appropriate class size is a 


complicated question that must be considered by learning organizations, which should consider 
(a) the type(s) of information being taught and (b) the technologies available to support the 
learning environment. One generally recommended “rule” is that class size should be guided 
by nature of the content (see Bloom’s taxonomy). Topics that require higher-level thinking (e.g., 
synthesis and evaluation) may be best suited to smaller class, whereas topics that entail lower- 
level thinking (e.g., recall) may be taught in larger classes (Taft, Kesten, & el-Banna, 2019). 


Traditionally, eLearning has been scaled up by offering multiple course sections. Taft and 
colleagues (2011) report that the most common recommendation is 25 students per class for 
online classrooms. However, the ideal class size remains an open question, which is likely 
influenced by available technologies, TA/grading assistance, faculty training, and class 
level/topic. Importantly, it has been estimated that online teaching requires 14% more effort 
(Tomei, 2006) compared to face-to-face teaching. Tomei (2006) further estimated that online 
class sizes should be only 70% of the size of an in-person class. Independently, Anderson and 
Avery (2008) derived a similar estimate of additional effort (14.5%) for online classes compared 
to in-person classes. 


For scaling up courses with large enrollment—from hundreds to thousands of students—simply 
increasing the number of sections (and thus instructors and TAs) becomes prohibitive. For this 
level of scale up, learning organizations must have courses that incorporate appropriate student 
interactions, and these tools must be ready before the course is launched. These needs require 
faculty to be fluent in the technology of the online course and adept at using the technology to 
engage students (Laws, Howell, & Lindsay, 2003). However, as noted above, such large classes 
may need to be limited to the lower levels of Bloom’s taxonomy, with a focus on familiarization 
and basic content knowledge. 


Student Social Support 

As online distributed learning technologies continue to advance and propagate, the potential for 
isolating students has been acknowledged as potential problem (Ludwig-Hardman et al., 2003). 
“Virtual” interactions and asynchronous environments may result in fewer opportunities for 
students to interact with peers in meaningful ways. 


To combat this problem and similar challenges, learning organizations must offer student 
support services and mindfully enable additional social structures. There are several categories 
of support, such as academic services (e.g., advising, library, financial, and admissions) and 
social services (e.g., student organizations, psychological services, placement services, and 
instructor support). These services interact with and build upon other essential factors, including 
students’ family framework, personal satisfaction, and perceived course relevance. All these 
elements play critical role in students’ decisions to persist or drop out of online courses (Park et 
al, 2009). 


Relationships between Participants and Resources 

Interaction between learners, teachers, content, and technology form a complex and 
interdependent learning environment (Anderson, 2003). Anderson’s (2003) original model 
outlined several proposed relationships between the student, the instructor, and the content. 
Dron (2007) applied the model to social learning and added a “group interaction” component. 


Thus, Anderson (2003) and Dron (2007) together outline the interrelationships and 
intrarelationships of four elements: students, instructors, content, and groups. 


There is strong evidence that organizations need to provide resources to support these 
interactions. Bernard et al. (2009) demonstrated that the strength of the student-instructor, 
student-student, and student-content relationships were related to student outcomes. 
Zimmerman (2012) observed a statistically significant relationship between the amount of time 
students spent engaging in online course activities and students’ weekly quiz grades, which 
provides evidence for the importance of student-content interactions. This effect indicated that 
students with moderate to high levels of interaction outperformed students with low levels of 
interaction (Bernard et al., 2009). Notably, student-instructor interactions seemed less impactful 
than student-student or student-content interactions. This pattern is significant when considering 
at scale course structures (e.g., MOOCs) where students tend to interact with content in diverse 
ways to satisfy their own goals rather than instructors’ goals (Emmanuel & Lamb, 2017; Ho et al., 
2014). 


Supporting Infrastructure from Adoption to Sunset 

Technological infrastructure support should be at the core of any learning organization. Modern 
infrastructure must transcend vertical and isolated systems to embrace open data formats that 
can integrate data from across the learning enterprise (Walcutt & Schatz, 2019). This idea is not 
new. In a 1996 paper on technology in learning organizations, Yohe (1996) described how 
organizations struggled to deliver new technology for users while also (a) maintaining legacy 
systems beyond their reasonable lifespans, (b) seeking interoperability between incompatible 
applications, and (c) doing so with dwindling resources. Angolia and Pagliari (2016) found that 
developing and sustaining quality distance learning programs required universities to possess a 
variety of supporting infrastructures. Such resources included appropriate policies and 
processes, information and communication technologies, instructional support staff, technology 
hardware and facilities, and training. Ricci (2002) similarly warned that successful institutions 
must have a comprehensive support structure in place for faculty, staff, and students with 
emphasis on technology support. 


Technological Practices 


Data Supported Courses 
To modernize courses and enable information sharing, learning technologies must be able to 
collect and output learning data. Several data standards are already in use. For example, xAPI is 


a popular method for capturing, standardizing, 

and sharing human performance data. “Data analytics is going to continually grow and 
become a critical part of our organization as 

Within xAPI, all learning experiences can be ASU EdPlus seeks to understand what's 

represented as interactions both internal and happening with all the students that we're 

serving. We are going to be innovative as far as 


the quality of education online.” 
James Cunningham, Senior Research Analyst EdPlus 


external to the online environment (Murphy, 
Hannigan, Hruska, Medford, & Diaz, 2016). 
These data can be stored within databases for 
later analysis via learning analytics and data mining techniques. The output of these analyses can 
then be used to optimize future learning through increased personalization (e.g., of learning 
materials or processes) or data visualizations (e.g., dashboards that offer feedback or 


recommendations to students, instructors, or administrators). Additionally, these data can be 
used to detect unproductive learning behaviors (Papamitsiou & Economides, 2014) and even 
cheating behaviors (Chuang Craig, & Femiani, 2017). Long and colleagues (2015) implemented 
personalization and visualization strategies within a rifle marksmanship course, resulting ina 
nearly 40% reduction in training time. Although this approach is promising, additional research 
is needed to determine the best practices for implementation and impact. 


Video 

Instructional videos have been a foundational and population resource for online learning 
environments and learning at scale (Davis et al., 2018). Asynchronous video (i.e., prerecorded 
videos that can be accessed outside of scheduled course time) is one of the most widely adopted 
technologies (Malaga & Koppel, 2017). Both students and instructors believe that video is an 
appropriate way to communicate course content (Miner and Stefaniak, 2018), and these 
perceptions are supported by evidence. Scagnoli, Choo, and Tian (2019) reported that video 
lectures were an effective means of delivering content, providing teaching presence, and 
enhancing student engagement in a virtual learning environment. 


Importantly, instructional videos can vary widely in quality and efficacy (MacHardy & Pardos, 
2015). Students learn better from videos that adhere to research-based principles of multimedia 
design. These principles enable learners to engage new material in ways that respect human 
cognitive capabilities (deKoning, Hoogerheide, & Boucheix, 2018), such as modeling successful 
learning behaviors (Craig & Douglas, 2019; Twyford & Craig, 2017) or providing dialogue 
interactions to ground procedural information (Craig, Chi, & VanLehn, 2009; Gholson, Coles, & 
Craig, 2010). Information on effective video creation can be found in Appendix B under Video 
and Appendix C under Social Cognitive Theory and Cognitive theory of multimedia learning 
(CTML). 


West, Armstrong, & Borup (2017) identified actionable strategies for implemented instructional 
video within online environments that can improve efficiency for instructors, make video 
personable, and make videos more effective teaching tools. To improve efficiency, they 
recommend writing out note of what will be said first, turning these notes into summary notes for 
students, and keeping videos short. To make videos more personable, they recommend 
projecting your personality (even if it more for unpolished videos) and be positive and 
conversational especially in early videos. To increase effectiveness, it was recommended that 
videos are concise, be aware of your setting and lighting, also use video feedback to give 
overall feedback to students. 


Virtual Reality, Augmented Reality, Gaming, and Simulations 
Although video is one of the oldest and most used formats, modern technologies have enabled 
the rapid rise of more sophisticated learning environments that simulate or enhance real-world 


phenomena. For instance, virtual reality (VR) environments immerse learners in simulated 
experiences that may mimic real-world experiences (e.g., a virtual tour). Augmented reality 
(AR) environments provide information or interactions that are “overlaid” the real-world (e.g., a 
digital heads-up display while piloting an aircraft). Simulations can also be provided in 2D or 
“desktop” versions, and games can infuse elements of ‘‘play” in these virtual, augmented, or 
simulated learning experiences. One of the most critical affordances of such learning 
environments is the ability to create virtual settings for learning that would otherwise be 


impractical (Correia et al., 2014) or unsafe (Patterson, Pierce, Bell, Andrews, & Winterbottom, 
2009). Collectively, we refer to these scenarios as virtual learning environments. 


Virtual learning environments can be an effective tool within modern learning ecosystems. Ina 
structured review of the learning at scale literature, Davis and colleagues (2018) observed that 
such environments were consistently one of the most effective categories with respect to 
beneficial learning outcomes and behaviors. However, although effective, these systems can be 
expensive and require extensive human resources to build (Fuentes, 2018). Moreover, some 
evidence suggests that they can replicate or reinforce human biases during training (Gamberini, 
Chittaro, Spagnolli, & Carlesso, 2015; Zipp & Craig, 2019), which can negatively impact efficacy 
(Zipp & Craig, 2019). 


Virtual environments are best used for domains that involve stable rather than dynamic content 

(i.e., concepts are known and do not require frequent updating), and domains that are not easily 

achieved within the real world (Alison, et al., 2013). As with most learning technologies, these 

environments should be supported by well-established learning methods (Shubeck, Craig, & 

Hu, 2016, 2016), such as modeling of expected 

behavior and appropriate use of feedback (i.e., just- “In order to increase learning efficiency, you 

in-time and after-action reviews). Systems should provide more personalized feedback to the 

also incorporate behavioral performance logging students and also you have to deliver that one- 

mechanisms (e.g., xAPI) that enable detection of and Ce wa mean Wet each seen sale . 
have their own needs addressed all the time in 

response to errors or biases exhibited by users (Zipp pea] time.” 


& Craig, 2019). Richard Tong, Chief Architect 
Squirrel AI Learning 


Social Media and Cooperative Learning 

Social learning acknowledges and leverages the social nature of all humans, who shape their 
realities by scaffolding prior knowledge with new information and experiences (Bingham & 
Connor, 2015). Social aspects of learning (e.g., cooperation, competition, knowledge sharing, 
and teamwork) can be facilitated within online learning ecosystems via social media platforms 
and related technologies. The facilitation of social learning via social media has emerged as an 
engaging and effective pedagogical tool (Martin, Martin, & Feldstein, 2017). 


First, social tools and technologies permit learners to engage in interactive learning. Within the 
ICAP framework (i.e., Interactive-Constructive-Active-Passive; Chi & Wylie, 2014), interactive 
learning entails students’ co-construction of new ideas—students are simultaneously building 
their own knowledge and the knowledge of others. Students work together to create and 
transform knowledge more successfully than they might do so alone. Research shows that social 
media tools can improve social interactions and engagement within MOOCs (Bingham & Connor, 
2015) and similar online platforms. Social environments allow students to express their prior 
knowledge about the domain, discuss their current understanding, give and receive feedback, 
and co-construct new ideas. 


Second, social technologies build social capital. In online spaces, social capital includes the 
relationships that are formed in distributed social networks and how those relationships facilitate 
action (Coleman, 1990). In an educational context, social capital further includes intangible 
relationships that exist between families, institutions, and communities, which may take the form 
of obligations or expectations that serve to aid or hinder academic success (Ho, 2019). In 


research on social learning environments, Venter (2019) found that informal collaborative 
activities exceeded the mandatory levels of engagement from LMS interactions required by 
course instructors. Several students sought out study groups even before engaging in an online 
learning experience, and their commitment to those groups was that of a “family” of learners. 
Moreover, these commitments were maintained throughout students’ enrollment in the degree 
program. 


Microlearning and Mobile Learning 

Microlearning is a learning approach that emphasizes small learning units and short-term, 
focused activities (Hug, Lindner, & Bruck, 2006; Lindner, 2007). Microlearning activities are 
typically less than five minutes in duration (Jahnke, Lee, Pham, He, & Austin, 2019). Evidence 
suggests that this approach can be more effective than traditional classrooms, with students 
exhibiting better learning outcomes and reporting increased perceived autonomy (Mohammed, 
Wakil, & Nawroly, 2018; Nikou & Economides, 2018). Mobile-based microlearning is a relatively 
new approach that enables microlearning via mobile devices (e.g., smartphones and tablets; 
Hug et al., 2006). Evidence in support of this mobile approach has been observed within both 
MOOC (Kamilali & Sofianopoulou, 2013) and corporate settings (Clark, Jassal, Van Noy, & Paek, 
2018; Goggins, Jahnke, & Wulf, 2013). 


Pedagogical Practices 


Communities of Inquiry and Increasing Presence 

Acommon critique of online learning, particularly in the early years of online course delivery, 
was that learners could feel isolated from their peers and instructor. To address and prevent 
these situations, researchers began investigating how to create a community of inquiry (Col). 
Any group of individuals who work together to create both personal and shared meaning 
through processes of critical thinking, discourse, and reflection can function as a Col (Garrison, 
2017; Rovai, 2002; Shahrtash, 2017; Thompson & McDonald, 2005). 


The Col framework suggests that there are three types of presence that can be fostered to help 


facilitate the establishment of a Col: cognitive presence, social presence, and teaching 
presence. Cognitive presence, although inherently difficult to foster and study (Duphorne & 
Gunawardena, 2005; Garrison & Arbaugh, 2007), refers to the extent to which an individual can 


use critical thinking to construct meaning in an online course (Garrison, Anderson, & Archer, 
2001). Ina sense, cognitive presence refers to how effectively the course helps the learner to 
manipulate the content in their own context. On the other hand, social presence refers to how the 
individual perceives the learning group’s cohesion, such as how well they can openly 
communicate and express themselves (Garrison, Anderson, & Archer, 2000). Finally, teaching 
presence refers to how well the instructor designs and facilitates the online course (Garrison et 
al., 2000), which is critical for facilitating both social and cognitive presences (Nagle & Kotze, 
2010; Shea et al., 2014; Tolu, 2013). 


Establishing an effective Col may be critical for distributed learning. Muljana and Luo’s (2019) 
review found that many Col-related constructs, such as a sense of belonging and the course 
design, relate to online student retention. The Col framework has been investigated in a variety 
of online settings, such as MOOCs (Kovanovic et al., 2018), synchronous and asynchronous 
courses (Rockinson-Szapkiw, 2012; Rockinson-Szapkiw & Wendt, 2015; Claman, 2015), and 


blended learning environments (Akyol, Garrison, & Ozden, 2009; Shea & Bidjerano, 2012). 
However, it is possible to support presence using asynchronous video, which can include short 
video lectures, video feedback, or learner response videos in discussions (Borup, West, & 
Graham, 2012). However, it is important for instructors to build a welcoming and professional 
space for these videos, and to be mindful that negative non-verbal cues (e.g., voice tone, 
posture, and facial expressions) can have a strong impact on learners (Thomas, West, & Borup, 
2017). 


Blended Learning 

Blended learning environments combine face-to-face learning with online learning or other 
forms of technology, although there is no clear-cut definition for the specific ratio of face-to-face 
and online opportunities that qualifies as “blended learning” (Graham & Dziuban, 2008; 
Millichap & Vogt, 2012; Stacy & Gerbic, 2008). Blended learning offers flexibility, ease of access, 
and the use of technology to enable learning. Moreover, students have been shown to 
experience an increase in creative thinking, tailored learning, and independent learning in 
blended learning settings (Becker et al., 2017). For such reasons, blended learning (along with 
mobile and online learning) have been described as a ‘foregone conclusion’—its use in 
educational settings, particularly colleges and universities, is on the rise (Becker et al., 2017). 


These claims have been consistently supported by empirical research and summarized in 
multiple meta-analyses. In a meta-analysis of forty-five studies, Means and colleagues (2013) 
found that students participating in online learning performed better than students receiving 
face-to-face instruction, and these improvements reached statistical significance when blended 
learning was the delivery mode. However, Means et al. (2013) also noted that blended learning 
studies generally increased learning time (i.e., time-on-task) and offered additional course 
resources as part of the instructional design. In a meta-analysis on blended learning, Liu et 

al. (2016) found that blended learning in health settings had a large, consistent, and positive 
effect compared to control settings. In a more recent meta-analysis, Dziuban, Graham, Moskal, 
Norberg, and Silicia (2018) found that blended learning improved success rates for most 
students, whether minority or non-minority. Students also ranked blended learning as their most 
preferred delivery mode. Researchers also observed that students in blended learning classes 
perceived course objectives and progress toward the objectives as important, along with 
enjoying an effective learning environment and communication from the instructor. 


Blended learning is an interesting hybrid of face-to-face and online learning. It is important to 
note that our original section on human learning principles still applies to blended learning. 
Additionally, many of the institutional and technological principles (see Learning at Scale) could 
also be useful for supporting blended learning. In the sections below, we review features and 
practices of learning institutions, technology, and pedagogy that are crucial for supporting 
blended learning. 


Institutional Practices 


Support for Blending Leaming Classrooms 
The success of blended learning requires institutional policies and plans for guiding the 
implementation of blended learning environments (Becker et al., 2017). These policies may 


include plans for faculty development, strategies for making necessary curricular changes, and 
financial appropriations to enable a switch to blended modes of delivery (Becker et al., 2017). 


Teacher training and support should include models of best practices in blended learning, along 
with exemplar courses, to aid instructors in (re)designing content (McGee & Reis, 2012). 
Teachers may be suspicious of vague directives issued by administrators. Blended learning 
initiatives can also induce stress for instructors who fear that course quality may decrease or that 
they will lose intellectual property rights in the transition (Moskal, Dziuban, & Hartman, 2012). 
Dziuban and Moskal (2011) reported that one successful training strategy was to offer faculty a 
professional development course using a blended format for an extended period (e.g., eight 
weeks or 80 contact hours). In this approach, faculty members become “the students” and 
experience the blended context for themselves. Training support for faculty has been shown to 
improve faculty satisfaction with teaching blended course sections. 


Moskal et al. (2012) stresses that institutions implementing blended learning must have a robust 
infrastructure that can handle continuous change. An understanding of blended learning 
strategies must be integrated throughout the academic system, including the registrar, teaching 
and learning centers, and technology centers for academic and IT concerns (McGee & Reis, 
2012). Without an understanding of blended learning, organizational support units will be 
unprepared to guide learners, and might even offer advice aligned to traditional classroom 
learning rather than blended learning. 


Class Size Recommendations for Blended Learning 

Blended learning can serve as a relatively fast and simple method for increasing the size 
capacity of face-to-face classrooms. Blended learning has been shown to support larger class 
sizes from 60 to 200 learners (Schell, 2012). Within large classes of comparable size, blended 
learning classes have also demonstrated higher student achievement compared to lectures 
(Deri, Mills, & McGregor, 2018). 


Technological Practices 


Limited Fvidence for Data Driven Learning 


Our review did not identify any studies that specifically delineated the effectiveness of 
dashboards and visualization techniques in blended learning contexts. Long and colleagues 
(2015) implemented this strategy within basic rifle marksmanship training and observed a 
nearly 40% reduction in time spent training. While this area is promising, additional research is 
still needed to determine the best practices for implementation and overall impact. 


Microlearning Principles and Mobile Learning in Blended Learning Environments 

Mobile learning has been shown to improve student participation, achievement, and learning 
within blended environments (Suartama, Setyosari, & Ulfa, 2019). Mobile blended learning is the 
use of mobile devices for learning. It is normally integrated within learning environments. 
Mobile internet technology has created opportunities for blended learning. Microlearning is a 
learning approach based on small learning units and short-term focused activities (Hug, Lindner, 


& Bruck, 2006; Lindner, 2007). They are normally less than five minutes in length (Jahnke et al., 
2019). 


To integrate mobile learning, Suartama and colleagues (2019) recommended a three-phase pre- 
analysis that (a) evaluates learners’ prior knowledge and characteristics, (b) employs learning 
object identification to determine what must be taught about the subject, and (c) analyzes the 
blended learning environment to select learning activities and resources, and to determine how 
assessments will be conducted. To identify design principles and essential characteristics for 
mobile microlearning platforms, Jahnke et al. (2019) conducted a review academic research 
literature, industry reports, and interviews with industry professionals. Eight major themes were 
identified for creating effective mobile learning content: Interactive micro-content for closing 
practical skill gaps, creating chunked courses, highlighting the instructional flow for activity- 
based model of instruction, system design (i.e., App availability, push notifications, track 
learning progress, searchable micro-lessons), supporting learner needs, Supportive social 
structures, costs, and curriculum provides single lessons but sum up into certificates/degrees. 


Video: Procedural Interactions and Modeling Behavior 

As with online learning, video is a popular method for blended learning. Many instructors view 
blended learning as just putting the lecture online. While video can provide useful material for 
students to engage with, it needs to be used as part of an overall blended learning approach 
(Mitra et al., 2010). Within blended learning settings, students respond positively to video 
communication and it has been shown to improved perceptions of instructor immediacy and 
social presence (Borup, Graham, & Celasquez, 2011). 


The overall use of video in blended learning is not different from the use of video previously 
discussed within online learning. As with video in micro learning content, it is best to keep video 
short. Learners often perceive long video segments as having poor alignment with other 
curricular learning activities and as less helpful (Lehman, Seitz, Bosse, Lutz, & Huwendiek, 2016). 
Also consistent to video-based learning in other areas such as online and face to face classrooms 
(Gholson, Coles, & Craig, 2010; Twyford & Craig, 2017), video within blended learning 
environments using modeling of a defined standardized procedural sequence, explanatory 
comments, and demonstration of infrequent procedures were perceived as most useful by 
students (Lehman et al., 2016). 


Pedagogical Practices 


Motivation and Self-regulation 

Varthis and Anderson (2016) found that blended learning environments increased learner 
motivation, learning skills gains, active learning, perceptions of learning quality, and student 
self-regulation. Van Laer and Elen (2017) observed seven attributes of blended learning 
environments that promote self-regulation: (1) authentic tasks; (2) tailored learning experiences; 
(3) learner control of pace, content, sequence, and learning activities; (4) scaffolding that helps 
students bridge their current zone of proximal development; (5) learner collaboration with the 
instructor and other students; (6) using cues to signal learners to reflect on critical content; and 
(7) calibration processes that allow learners to evaluate their own performance. The researchers 
suggested that blended learning may prove more challenging for less self-regulated students 
than for highly self-regulated learners. However, Silva, Zambom, Rodrigues, Ramos, and de 


Souza (2018) observed that providing learning analytics feedback at frequent intervals improved 
student self-regulation in blended learning environments. 


Competency-Based Leaming 

Competency-based learning (CBL) is parsing of learning into specific chunks of skills and 
knowledge. It involves the creation of learning outcomes to clearly establish levels of mastery 
and assessments that allow learners to demonstrate their mastery. It is more output driven with a 
focus on the learner and the learning (Stafford, 2019). 


The Department of Defense is responsible for training and educating personnel to a minimum 
level of proficiency. Traditionally, there has been a separation between these two entities. 
“Education” has typically emphasized incremental and gradual gains in conceptual 
understanding, whereas “training” has emphasized readiness, demonstrable skills, and 
immediate feedback (Smith, Hernandez, & Gordon, 2019). In addition, CBL is less common in 
education than in training because of the difficulty of extrapolating competencies from purely 
cognitive development (Stafford, 2019).CBL accounts for the unique training that occurs in 
military contexts that encompass the service members knowledge, attitude, skills, traits, 
abilities, and other aptitudes (Smith et al., 2019). Successful implementation of CBL can be 
attained by providing user-friendly, real-time mapping tools to help guide curriculum (Wong et 
al., 2019). Maza, Lozano, Alarcon, Zuluaga, & Fadul (2016) found that students gained flexibility 
and autonomy in the learning process and were able to develop cognitive, procedural, 
technical, integrative, professional, communicative, and reflective competence. 


Although there have been decades of interest in implementing competency-based, our review 
found limited evidence for effectiveness within blended or eLearning environments. Previous 
reviews have obtained the similar results and have called for strong empirical quantitative 
evidence of pedagogical effectiveness (Henri, Johnson, & Nepal, 2017). This is not a 
recommendation that competency-based learning should be avoided or abandoned. However, 
empirical research on competency training, both at scale and in blended learning environments, 
is still lacking (see Appendix C: Competency-based Learning for more detailed discussion). 
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General Methods Section 


literature Review Methodology 

To locate and synthesize relevant literature for this review, we used a broadly scoped review 
process consisting of more than 200 formal database searches. Our search strategy was aimed at 
trying to locate resources from academia, the military, or industry when relevant, and included 
the following databases: Academic Search Complete, Academic Search Premier, ACM Digital 
Library, DTIC, ERIC, Google Scholar, Education Full Text (H. W. Wilson), Education Research 
Complete, IEEE Xplore Digital Library, I/ITSEC, Military and Government Collection, ProQuest, 
NDIA Repository (2018 Proceedings), PsychInfo, PubDefense, SAGE Journals, Scopus, Sports 
Medicine and Education Index, SpringerLink, Teacher Reference Center, and Web of Science. 


We kept records of the database searches conducted, the relevant keywords, and databases 
used in these searches. After conducting a search, we first reviewed the titles and abstracts of 
articles to determine their relevance to this report. When articles were deemed relevant, they 
were set aside for full-text review. If the article was still found to be relevant after reviewing the 
full text, it was set aside to be potentially included in the relevant research summary. Member(s) 
of our team reviewed relevant studies located and synthesized them into the summaries present 
in this report. 


Survey Methodology 
Design/participants and procedure 


A survey was conducted to determine integration of the best practice within learning 
organizations and the views held by individuals within these organization of the practices. The 
survey data was collected entirely online through Qualtrics. Participants were recruited from 
organizations that specialized in eLearning, including academic (e.g., college, school, or 
university), private (e.g., company or industry), and public (e.g., military or government) 
sectors. All participating organizations were required to have headquarters based in the United 
States. The recruitment process consisted of contacting (i.e., email and phone) 105 
organizations. During this initial contact phase, the purpose of the study was explained and a link 
to the online survey was provided. The contact phase consisted of five rounds for each 
organization; two rounds to establish contact, a round of sending the email containing the link to 
the survey, and two rounds of follow up emails to help remind participants to fill out the survey. 
Out of the 105 contacted organizations a total of 16 responses were received, which consisted of 
a 15.24% response rate. 52 academic organizations were contacted and 6 completed the survey 
at a response rate of 11.54%, from the 45 private organizations contacted 6 completed the survey 
at a response rate of 13.33%, and out of the 8 public organizations contacted 4 completed the 
survey at a response rate of 50%. 


The survey itself contained a demographic questionnaire (e.g. age, education, organization 
type), four question categories, and an open-ended response question. The age of participants 
ranged from 29 to 66 (M = 48.29, SD = 10.62). The educational levels held by the participants 
were 5 Bachelor’s degrees, 7 master’s degrees, and 4 Doctorates. The responses received from 
each organization type consisted of 6 Academic (42.85%), 6 Private (42.85%), and 4 Public 
(25%) institutions. This report outlines significant differences found between responses from the 
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military run organizations (25%, the public organizations) and civilian run organizations (75%, 
the academic and private organizations). 


Survey description 

The demographic questionnaire asked age, gender, race, highest level of school completed and 
field of the degree, current position within their organization, time held in their current position, 
time held in their current organization, overall description of their organization (Academic, 
Private, and Public), and how they would describe their learning organization. 


Following the demographic questionnaire, the survey was divided into four categories of 
questions with each question using a 6-point Likert scale for participants to rate their perceptions 
of their organizations in several key e-learning areas. Participants were asked to rate how 
important a certain e-learning feature was to their organization and how well that feature was 
incorporated in current practice. These paired question categories explored an organization’s 
use of technology, technological features, instructional methods, and supporting principles. 


e Use of Technology —- Perceptions on the use of technology (e.g., intelligent tutoring 
system or video) during learning 
Technological features — Perceptions of features (e.g. personalization) that technology 
can offer 
Instructional methods — Perceptions of instructional methods (e.g., at scale, blended 
learning, synchronous eLearning) for supporting learning 
Supporting principles — Perceptions of principles of learning (memorization, 
collaboration) 


Finally, there was an open-ended question in which the participants were asked what they would 
recommend their organization do to better support eLearning. 


General findings 


Results for this survey generally reinforce the known gap between research and practice. The 
survey found that reported use fell just slightly above the midpoint indicating only slight 
adherence. However, perceptions of best practices were consistently higher with respondents 
agreeing on importance. Detailed findings and tables are available in Appendix D: Survey 
Results. 


Appendix A: Institutional Systems Review 


Transitioning more traditional classroom-based (i.e., face-to-face) learning organizations into 
modernized learning organizations that utilize advanced technological learning techniques is 
not a simple task. However, the good news is that some of the best practices are like those of 
traditional learning organizations but need support for a transition into an online medium. 
Thus, learning organizations must have commitment to technological infrastructure, human 
infrastructure, and human centered design focus (Walcutt & Schatz, 2019).When thinking 
through the new learning organization structure, there must be clear understanding of the 
resources at hand to set up the organization, an understanding of the members (i.e., 
stakeholder groups) within the organization and their needs, as well as the process by which 
the organization will function and policies that will govern the organization (Giattino & 
Stafford, 2019). Specifically, Rovai and Downey (2010) state that the factors that lead 
online/distance learning organizations to fail are planning, marketing and recruitment, 
financial management, quality assurance, student retention, faculty development, and online 
course design and pedagogy. 


State-of-the-art distributed learning environments use learning principles and 
strategies and are supported by a network of learning specialists. 

State-of-the-art distributed learning environments work to establish trust at all levels 
establishing them using transparency when possible. 

State-of-the-art Distributed learning environments provide a full framework of student 
social support. 

State-of-the-art distributed learning environments provide an integrated institutional 
support system that focuses on interactions as a key element. 

State-of-the-art distributed learning environments plan and allocate resources for 
technology support and training from adoption to sunset. 

State-of-the-art distributed learning environments provide sufficient support for 
blending learning classrooms. 

State-of-the-art distributed learning environments have flexible class sizes based on 
needs and provide adequate technology for supporting larger class sizes. 


Understanding the learning process 
A Holistic Approach to Education - Pedagogy, Andragogy, and Heutagogy 


For generations, educators have practiced the art and science of learning and teaching, called 
pedagogy (Bandura, 2005). In pedagogy, instructors use standard teaching strategies intended 
to target all learners, who are considered receptacles (Crawford, Young Wallace, & White, 
2018). The learner is passive and dependent, while the teacher’s goal is to pass on knowledge 
and culture to the students (Bangura, 2005) for the purpose of changing, shaping, or controlling 
behavior (Knowles, Holton, & Swanson, 2015). The job of the teacher does not end after they 
impart the required knowledge, because teachers are charged with teaching students how to 
learn and think critically (Bonney & Sternberg, 2017). 


For the teacher or the learner to be considered effective, some assessment must be made on the 
process and outcome (Hattie, 2009). Pedagogy is framed around learners assimilating the goals 
and rules of a subject in a process called single loop learning (Crawford et al., 2018). The goal of 
single-loop learning is the use of knowledge to avoid mistakes in an actions-results approach. 
Learning design often takes on a linear format as the learner moves from one element of the 
subject to the next. Consequently, modular learning is often used in pedagogical designs 
(Crawford, 2018). 


Pedagogy may assume a “blank slate” approach to learning which adequately reflects a 
student’s lack of prior knowledge and understanding, however not all learners are 
inexperienced or young, and more mature learners may require a modified educational 
approach (Crawford et al., 2018). The term andragogy was coined to speak to the way adults 
learn and how best to teach them (Bangura, 2005; Crawford et al., 2018; Knowles et al., 2015). 
Andragogy, as envisioned by Knowles et al. (2015), places the adult learner in a more active role 
of deciding what they need to know, why they need or want to know it, and how to go about 
learning it. Andragogy lifts learners out of their dependency on the instructor and makes a 
learner’s own experience and motivation central to learning. Andragogy esteems learner- 
directed plans and activities, offers mutual control between the teacher and the student, and 
demands a collaborative atmosphere where the learner looks to the instructor to bring a new 
character to the knowledge (Crawford et al., 2018; Knowles et al., 2015). According to Crawford, 
and colleagues, andragogy uses a double loop learning process that is framed around 
formulating a deeper understanding of a subject so that the learner can look beyond the actions 
or results, and function more adroitly than just avoiding mistakes. Double-loop learning helps 
learners use knowledge proactively and form a deeper understanding that results in a better 
ability to derive meaning from the acquired knowledge. Working with knowledge in new and 
different ways adds depth of understanding to the knowledge base, which results in a sort of 
spiral or cyclical learning where knowledge re-organization is scaffolded into higher order 
thinking Some learners are more self-directed and self-determined. These learners possess an 
awareness of the subject matter and have already decided what and how they would like to 
continue learning about the subject. Learning experts developed a third learning framework 
called heutagogy to explain this self-directed learning paradigm (Crawford et al., 2018). The 
heutagogy framework operates on the principles of andragogy with an enhanced focus on 
learner autonomy (Blaschke, 2012; Crawford et al., 2018). Self-directed learners can accurately 
choose what knowledge they are lacking in a subject and pursue improved understanding in 


ways that are most relevant to their situation (Crawford et al., 2018). These learners can use 
multiple engagement styles in what is called a triple-loop learning style, where knowledge is not 
just mistake-avoidant and proactive, but is also transformational to the point of mastery learning 


due to the progressive reflection on the subject matter (Crawford et al., 2018). Also, as learners 
gain independence, the role of the instructor diminishes (Canning, 2010). 


In comparing pedagogy, andragogy, and heutagogy, one should consider these methods based 
upon their nature, focus, power structure, design, attention to the learner’s perspective, and the 
ability to foster learner development (Crawford et al., 2018). Below is a table summarizing the 
differences in these educational foundations. Crawford et al. (2018) suggest that instead of 
viewing pedagogy, andragogy, and heutagogy as some type of age-related phenomenon, these 
constructs should be viewed as a progression within a learner’s individual learning pathway in 
any given subject (Crawford et al., 2018). In this scenario, learners may begin learning a subject 
from a pedagogical perspective which is teacher-directed and knowledge-based, and progress 
through a more andragogical stage in which the teacher and student coordinate learning. 
Finally, as a student begins to approach mastery, they progress to a more self-directed, 
autonomous plateau in which they control more of the learning process (Blaschke, 2012; 
Crawford et al., 2018). Blaschke (2012) claims that heutagogy should be given serious 
consideration in the current educational climate because of its net-centricity, and that it could 
serve both distance and traditional educational paradigms in a time of emerging technology. As 
a modification of andragogy, heutagogy shares the same audience and goal of making self- 
sustaining learners. Further, a heutagogical approach could also be beneficial for preparing 21st 
century learners for pursuing multiple career paths or re-skilling throughout their lifetime. 


Characteristics 


Learning Style 


Focus 


Power/Control 


Learning Progression 


Design 


Learner Development 


Comparison of Educational Frameworks 


Pedagogy 


Instructor Directed 


Knowledge 
Acquisition 


Instructor Directed 


Single loop (rules, 
objectives) 


Linear or modular 


Prerequisite 
Knowledge 


Andragogy 


Self-Directed 


Content 


Instructor/Learner 
Directed 


Double loop 
(modifying 
application) 


Cyclical or Spiral 


Competency 
Development 


Heutagogy 


Self-Determined 


Process 


Learner Directed 


Triple or Spiral loop 
(transformative) 


Holistic/Mastery 


Capability 
Development 


es (McLoughlin & Lee, 2010). 


(Table contents from Crawford et al., 2018). 


Blaschke (2012) found that one possible caveat is that some campus-based students may exhibit 
less maturity and possess less prior experience than working adults and that Web 2.0 and social 
media may be suited to heutagogy by encouraging individual learning experiences due to these 


platforms’ encouragement of user-generated content . Students need a rich environment for 
learning that is both social and participatory, and instructors are intrigued by the opportunity to 
support learning in these diverse environments (Dron, 2007; McLoughlin & Lee 2010). Social 
software, by its nature, seems to recommend heutagogy because it encourages meaning- 


making, engagement, and collaboration (Dron, 2007). 


Interaction between learners, teachers, content, and technology, form a complex 
interdependent learning environment (Anderson, 2003). Anderson’s (2003) original model 
outlined the proposed relationships between the student, the instructor, and the content. Dron 
(2007), to hone the model to social learning, added a group interaction component. 


Bernard et al. (2009) demonstrated that the strength of the student-instructor, student-student, 
and student-content relationships were related to the effect size of student outcomes, thereby 
supporting Anderson’s (2003) model. Zimmerman (2012) found a statistically significant 
relationship between the amount of time students spent engaging in online course activities and 
the student’s weekly quiz grades, which provides further evidence for the importance of the 


student-content interaction. The effect size was evident with both moderate and high student 
interaction levels compared to students with low levels of interaction (Bernard et al., 2009). 
Further, they found that, overall, student-instructor interaction treatments were less impactful 
than student-student or student-content interactions. This may be a significant finding when 
considering certain course structures, such as MOOCs, where students tend to interact with 
content in diverse ways to satisfy individual goals (Emmanuel & Lamb, 2017; Ho et al., 2014). 


No discussion of teaching or learning can be cogent without reflecting on the human cognitive 
architecture and its limitations, especially in regard to working memory and its constraints and 
specifically in the context of the novice learner (Hattie, 2009; Paas & Sweller, 2014). Further, no 
discussion of teaching and learning would be complete without acknowledging the influence 
(whether negative or positive) of the individual learner, the home, the school, the curriculum, the 
teacher, and the instructional approach (Hattie, 2009). It is here that the amalgamation of 
institutional policies, governance, support, and student characteristics have their interplay in 
shaping the learning experience and outcome. 


The act of teaching and learning does not take place in a sterile environment, nor can either 
endeavor take place automatically (Hattie, 2009). Hattie suggests that the hindrances to effective 
teaching and learning are numerous, and the pinnacle of success in teaching and learning 
happens “next”; after the information has been structured, designed, imparted, interpreted, 
accommodated, reacted to, and applied. He describes learning as individualistic, spontaneous, 
effortful, often slow, and gradual, and moving forward in the manner of an old jalopy, -in fits-and- 
starts. Learning and teaching are inseparable, and each participant requires the other to exhibit 
effort, attention, patience, and passion. 


Foundational elements of online course design 
Classroom Management 


Online learning can be delivered in multiple formats such as blended courses, where there is 
some combination of classroom and online interaction, synchronous online learning, in which 
there is some set course time for course instruction, and asynchronous online learning, which 
allows students flexibility for interacting with course materials (Bernard et al., 2004). An early 


meta-analysis performed by Bernard et al. (2004) revealed that, in online learning environments 
to that point in time, mean achievement effect sizes favored the classroom form of instruction 
over synchronous learning and asynchronous distance education was favored over classroom 
instruction, although the authors warn that there was too much heterogeneity in studies to make 
distinct recommendations. Furthermore, it should be noted that this study was published more 
than 15 years ago. 


For classes taught via distance education, whether asynchronous or synchronous, certain 
practices are recommended, such as paying specific attention to quality course design instead of 
media characteristics. Active learning, such as problem-based learning with some form of 
collaboration, is encouraged to make distance education courses profitable for deep learning. 
Other recommendations include pre-recorded video, some form of “face-to-face” interaction, 
providing information about courses in advance, and interactivity in media (for asynchronous 
distance education classes) (Bernard et al., 2004). 


In a more recent meta-analysis of 45 studies, Means, Toyama, Murphy, and Baki (2013) found that 
students participating in online learning performed better than students receiving face-to-face 
instruction and that the improvement reached significance when blended learning was the 
delivery mode. Means et al. (2013) note that blended learning studies generally reported 
increased learning time and additional course resources as part of the instructional design. 
Further, the blended learning studies used design elements that promoted learner interactions 
(Means et al., 2013). In a meta-analysis of the effect of blended learning in health professions, Liu 
et al. (2016) found that blended learning had a large consistent positive effect compared to no 
intervention in health professions learning. Additionally, blended learning courses 
outperformed non-blended courses, demonstrating that, in health professions learning, blended 
courses are more effective for student learning (Liu et al., 2016). 


Yet, not all online or blended courses are taught in the same way. Martin, Ritzhaupt, Kumar, and 
Budhrani (2019) identified online faculty that were acknowledged by the Online Learning 
Consortium, the Association for Educational Communications and Technology, or the United 
States Distance Learning Association to query these esteemed professionals on their online 
design processes. These authors noted that the recognized faculty recommended using 
systematic design processes, including chunking meaningful content, backwards design 
processes, ascertaining learner’s needs, and designing learner interaction into their course 
designs (Martin et al., 2019). Student engagement was maintained through timely responses and 
feedback, periodic communication about the course, and demonstrating instructor availability 
and presence (Martin et al., 2019). Further, Martin et al. (2019) found that accomplished online 
instructors incorporate a variety of assessments into their courses and used rubrics to steer 
student evaluation. These faculty also paid attention to course feedback, learning analytics, and 
peer assessment to improve their online offerings (Martin et al., 2019). Other suggested 
practices to aid in course management aimed at retention include making financial assistance 
available, providing counseling and library services, providing prompt feedback, providing 
opportunity for students to learn technology skills, making student assignments with social 
interaction, utilizing diverse approaches to student engagement, and ensuring reasonable 
expectations of student performance by identifying success factors for the class (Aversa & 
McCall., 2013). 


In the case of MOOCs, due to their open and less-regulated format, it is advisable to support 
learners in becoming self-regulated through instructor interventions such as prompts, feedback, 
and using learner analytics to tailor support for individual students (Wong, Baars, Davis, Van Der 
Zee, Houben, & Paas, 2019). 


Retention in Online Courses 


Recently, a systematic review by Muljana and Luo (2019) listed strategies to aid student retention 


as a component of course management. At the institutional level, support for student retention 
includes communication, orientation to online learning, and adequate student support services, 
including technical support (Angelino, Williams, & Natvig, 2007; Aversa & McCall, 2013; Bunn, 
2004). Early measurements of student participation in the course have been shown to be 
predictive of course completion, such that instructors can identify low participation learners and 
intervene to provide support (Nistor & Neubauer, 2010). Further, Boston, Ice, and Gibson (2011) 
found that ‘‘swirling,” a term used to denote students who purposefully attend at least two 


colleges or universities prior to graduating, may affect an institution’s perception of retention. 
Student support services are also cited as being demonstrably beneficial for student retention 
(Nichols, 2010). 


Course Quality and Accessibility 

Quality course design is critical to making online courses available to all types of learners 
(Martin, Ndoye, & Wilkins, 2016). There are several outside organizations that are available for 
assisting educators and developers in the quest for online course excellence. Quality Matters 
(QM) is a subscription service for online course developers that provides them with the highest 
standards for designing online courses (Loafman & Altman, 2014). These authors state that QM is 
built around a strong research base and users follow a rubric to evaluate their courses for online 
accessibility and student support. The QM rubric addresses eight aspects of pedagogy that, 
working together, can improve online course offerings (Martin et al., 2016). These 
characteristics address the inclusion of a course overview and introductory materials, writing 
and following course learning objectives, designing effective assessment and measurement of 
students, providing instructional materials, selecting course activities and maintaining learner 
interaction, utilizing course technology, providing learner support, and designing online 
materials with accessibility and usability in mind (Martin et al., 2016). Dietz-Uhler, Fisher, and 
Han (2007), reported a 95% retention rate when using QM in online course design, although 
Fredriksdottir (2018) reported that retention rates between 2.4% -18.2% depending on the 
course delivery method. 


Loafman & Altman (2014) suggest other resources for developers to consider including The 
Online Learning Consortium (formerly The Sloan Consortium) is another resource center for 
online course developers to access content on best practices in designing online learning. A 
third quality-focused group is the Texas Higher Education Coordinating Board which developed 
the “Principles of Good Practice” document to guide online education. 


Personnel Requirements in Online Learning 

One of the first decisions to be made in beginning a distance learning program or course is who 
will be responsible for overseeing the development of the learning environment. Some 
institutions choose to organize and develop an online offering internally, while others choose to 
utilize an online program management provider (OPM) to take responsibility for the onboarding 
(Springer, 2018). The for-profit nature of the OPM allows the provider to invest some or all of the 
investment capital up-front to develop and launch the online program in exchange for a share of 
the profit that the program generates (Springer, 2018). Hillman and Corkery (2010) state that 
even institutions that are not novices in the distance education arena may discover that the 
university infrastructure may not be adequate to design and implement the online learning 
solution, which requires a necessary impingement on academic and non-academic departments 
(for example, the admissions offices and business offices). Out-sourcing does not have to be all- 
inclusive, as some institutions may have certain strengths departmentally, which can handle the 
overload of phasing-in distance learning (Hillman & Corkery, 2010). 


Whether utilizing in-house course development or an OPM, a needs analysis will need to be 
performed (Hillman et al., 2010). According to Khedhiri’s findings, measuring the institutions 
readiness to change can help planners and developers understand the climate of the institution 
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so that change is viewed as a solution to faculty demotivation, communication challenges, and 
teamwork issues on a personal level. Times of upheaval in institutions can have stakeholders 
looking to leadership for qualities that reinforce an alignment of individual stakeholder values to 
the institution’s goals. Leadership alone cannot prevent demotivation to institutional change nor 
the challenges that accompany the changes (Khedhiri, 2018). 


Hillman & Corkery (2010) found that when examining institutional readiness for launching an 
online education opportunity, there was duplication of services in some areas which required 
admissions offices, technology teams, bookstores, financial aid offices, and business offices to 
have representatives willing to help streamline processes for students to move quickly through 
those services. The solution these authors recommended was to have the processes so 
thoroughly streamlined that the students can move efficiently through all departments to provide 
excellent customer service with the goal of improving retention. Throughout the development 
process, stakeholders must hold on to a transition mindset with continual collaboration and 
communication 


The undertaking of distance learning by an institution is weighty and the final end-user, the 
student, must not be forgotten. Administrators must be mindful of their obligation to assist 
student learning, to bridge the gap between the instructors and the students, and to aid students 
in completing their program of study (Stein & Anderson, 2017). 


In summary, to develop competitive online courses, institutions must gather a team of 
instructional designers, subject matter experts, instructors, support staff, administrators, and 
learning engineers (Kurzweil & Marcellas, 2019). Other insights suggested were that learning 
engineers should function beyond a traditional instructional designer as they work in theoretical 
realms of education and learning, but also in analysis of data and interdisciplinary roles to bring 
learning professionals together to design and implement the learning ecosystem. Learning 
engineers can use data and analytics to scale learning using practical and theoretical models. 


UX Considerations 

Learning experience design has grown from origins in user experience (UX) to encompass the 
objectives of sound instruction, such as learner-centered design principles, usability, and 
interaction in the learning space (Schatz, 2019). Santoso, Schrepp, Isal, Utomo, and Priyogi 
(2016) have worked to establish a User Experience Questionnaire (UEQ) to address the major 
components of a user’s evaluation of the distance learning experience. These components 
include the attractiveness, efficiency, ease of use (perspicuity), dependability, stimulation, and 
the novelty of the experience. However, a recent study by Lallemand, Gronier, and Koenig 
(2015) found that defining a ‘good’ user experience may be difficult due to important differences 
between what is pleasing to people in different geographical locations and different cultural 
backgrounds. While nearly 84% of respondents to a UX survey stated that UX was central to their 
professional work, interest in UX was less central for responders who were researchers (whose 
primary interest in the topic was as a field of study) or students than for managers. The 
respondents’ definitions of what UX is depended on work domain (industry versus academia), 
and different cultures and levels of expertise affected the perceptions of whether UX definitions 
should be standardized. Most seasoned practitioners were less disturbed by stringently defining 
UX, which, they reasoned, was due to experts developing a working definition for themselves 
and no longer needing a shared viewpoint. Further, they also noted that respondents believed 
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UX to be an individualized notion, but when queried whether people could have a comparable 
UX definition, respondents were divided in opinion. 


Yet, experts agreed that UX is contextual (Lallemand et al., 2015). User experience in 
instructional design can be learner-centered, not just regarding focusing on the content and 
learning outcomes but viewing UX as a cooperation with learners to achieve the goal of learning 
(Matthews & Yanchar, 2018). Also, to truly make designs learner-centered, they recommend that 
instructional designers should invite learners to engage with content through meaningful and 
relevant instruction. This happens when designers imagine what learners would think and feel as 
they navigate the content areas to determine if the design is likely to be a favorable experience 
for the learner. If learning is truly a personal meaning-making, then designs should provide a 
suitable environment for that to happen (Clinton, 2015). 


It is important to think about students in terms of the various burdens placed on them in online 
learning, such as the overabundance of resources, which can be contrary to productivity (Shatz, 
2019). In a fast-paced, media-rich environment, students can suffer from inefficiency and 
ineffectiveness, which can diminish attention span, encoding, and decision-making (Schatz, 
2019). To counteract the rush and breadth of learning opportunities, instructional designers must 
consider the learner holistically and tailor learning to personalize it (Schatz, 2019). Further, 
helping students steadily improve in self-regulation abilities can improve their resistance to 
distraction, which not only improves concentration but assists with long-term encoding and 
decision-making (Schatz, 2019). 


Learner experience design tries to solve one of five problems for learners: a lack of knowledge, 
a lack of skill, a lack of motivation, a lack of confidence, or a lack of tools or resources for 
learning (Interaction Design Foundation, 2017). Of all these deficits, overcoming a lack of 
motivation is the most difficult to solve using learner experience design (Interaction Design 
Foundation, 2017). 


By thinking of the learners in terms of what is meaningful and relevant to them, UX designers 
may improve motivation (Interaction Design Foundation, 2017), however motivation has many 
complicating factors such as the self-directedness of the learner. According to the Interaction 
Design Foundation (2017), three questions can guide the design of an excellent learning 
experience. These are: 


1) What does someone need to know to do this? 
2) What does someone need to be able to do to complete this? 
3) What resources or tools are needed to do this? 


The answers to these questions determine what type of content is needed and how to best design 
and deliver that content (Interaction Design Foundation, 2017). Instructional designers must 
move from only considering the appropriate means and method of content delivery through a 
course or training unit, and move into considering lifelong learners who, with diverse 
experiences and contexts for learning, require more active and self-directed experiences 
(Bannan, Dabbagh, & Walcutt, 2019). This new paradigm will force instructional designers to 
imagine how learners think, feel, sense, act, and relate (Schatz, 2019). Furthermore, learner 


experience design will be tasked to multiple disciplines in addition to instructional designers, 
such as: learning scientists, engineers, and data scientists (Schatz, 2019). 


Student Characteristics, Barriers, and Support in Online Environments 

Instructor and student support in online learning is critical to minimizing attrition (Park & Choi, 
2009). Individual student characteristics, internal factors, and external factors can contribute to a 
student’s desire to complete or drop an online course (see table below) (Park et & Choi., 2009). 
Bell and Federman (2013) found that there was a higher dropout rate for students in 
asynchronous learning modalities and a more negative student attitude in synchronous learning 
environments, although there was no difference in overall achievement between traditional 
students and e-learners. 


Park & Choi (2009) found that students who dropped out of a course had perceptions that were 
significantly different in terms of internal and external characteristics when compared to 
students who persisted in courses. The learner’s framework of family, organizational support, 
satisfaction, and course relevance plays a critical role in their decision to persist or drop out of 
online courses, while age, gender, and educational level were not predictive. 


Individual Characteristics Age, Gender, Educational Level 
Internal Characteristics Family, Organizational Support 


External Characteristics Motivation (Satisfaction and Relevance) 


Adult learners are more likely to persist in their online courses when they perceive they are 
supported by their family and friends, and they are more likely to persist when they perceive 
that the learning organization supports them adequately (Park et al., 2009). I Initially, the 
organization should support the learner by maximizing the external characteristics of the course 
(satisfaction and relevance) to maintain student motivation, but that a shift to include maximizing 
student support should begin after the course is underway. Furthermore, they stated that 
instructional designers can facilitate this shift by planning and implementing their designs such 
that students can be encouraged when family support lags. 


Student Barriers to Online Learning 

Several researchers have examined student barriers to online learning. For example, 
Muilenburg and Berge (2005) found that there were eight factors that acted as barriers to online 
learning from the student’s perspective. These were administrative issues, social interactions, 
technical skills, academic ability, time and support in studies, student motivation, technical 
problems, and cost and access to the internet. The four most critical barriers were social 
interaction, administration or instructor issues, learner motivation, and time and support for 
studies. The variables with the largest effects on these barriers were a student’s ability and 
confidence with online learning technology, their effectiveness with online learning, their 
enjoyment of online learning, the number of online courses completed, and the likelihood of 
taking future online courses 


Sun, Tsai, Finger, Chen, and Yeh (2008) found that learners’ satisfaction was significantly and 
negatively affected by anxiety over computer use, and significantly and positively affected by 
the instructor’s attitudes about e-learning, course flexibility, course quality, perceived 
usefulness and ease of use of the computer learning system, and the diversity of learning 
assessments. Additionally, Sun et al. (2008) found that there was no significant effect of students’ 
perceptions of satisfaction related to their attitude about computers, their internet self-efficacy, 
the timeliness of an instructor’s responses, the quality of the technology, the internet quality, or 
the learner’s interaction with others. 


Park and Choi (2009) demonstrated that learners were more persistent in online learning when 
they were experiencing satisfaction with the course and when they see the relevance of the 
course to their lives. The importance of satisfaction and relevance are echoed by Yang, Baldwin, 
and Snelson (2017), who found that interest in technology, career goals, time and effort invested, 
and the perceived utility of the material were the individual attributes that led to persistence in 
online learning at a personal level. From an institutional perspective, course relevance to either 
individual or professional needs, course satisfaction, program satisfaction, and a connection 
between the course of study and a job promotion proved to be the most relevant factors 
influencing learner persistence (Yang et al., 2017). Muilenburg and Berge (2005) stated that 
since social interaction was the most relevant impediment to online learning, and because social 
interaction was strongly associated with online learning enjoyment, online learning 
effectiveness, and the likelihood of pursuing another online class, it would follow that improving 
social interaction would be a worthy goal for creating enjoyable, effective, and desirable online 
courses. 


Again, as with instructor support, it is necessary to address the unique barriers that students may 
experience ina MOOC. Ina study of student engagement in MOOC environments, Hew, Qiao, 
and Tang (2018) found that the most mentioned factors in engagement were instructor attributes. 
Students perceived the xMOOC format, a more traditional structured course with a syllabus, 
objectives, assignments, evaluations, etc. (Touro College, 2013), more favorably than the 
cMOOC, aMOOC formed with connectivist theories in which students and the instructor share 
responsibility for content and discussions (Hew et al., 2018). Since xMOOCs frequently have a 
video component, students also identified feeling engaged when the instructor used humor in 
the video. Other impactful engagement tools were identified by MOOC participants such as 
using real-world problems and solutions, content depth and difficulty, and interaction with and 
support from instructors or tutors. Further, this study found that MOOC students did not attach 
significance to relating to other participants compared to face-to-face courses or traditional 
online courses, perhaps due to the anonymous nature of the MOOC or the personal 
responsibilities of the participants. Knox (2014) found that students in cMOOCs considered the 
student submitted creations as superfluous or excessive and that a significant proportion of 
MOOC participants did not value peer contributions. Furthermore, the constructivist MOOCs, 
which offer a learner-centered experience, were found by students to be overwhelming and 
confusing and students frequently opined that guidance was lacking, and courses were lacking 
support. Dissenters to more course guidance in MOOCs felt that the learning could happen 
without an instructor, but in the absence of instructors the community must be fostered in some 
way. As shown, a variety of factors can become barriers to students enrolled in MOOCs. This is 


important, as Fridriksdottir (2018) found that all modes of delivery of MOOCs show low retention 
rates as low as under 5% completion. 


Students with disadvantaged backgrounds, especially minority students, and those with 
insufficient academic achievement, face unique challenges getting access to advanced 
education or work training (Deming, Goldin, & Katz, 2013). Often these students resort to for- 
profit colleges that rely heavily on federal grants and loans. For-profit colleges can be more 
expensive for degree programs than their community college counterparts, which further 
exposes low-income students to higher debt. For-profit graduates are more likely to be 
unemployed after the completion of the degree than those students graduating from community 
colleges and other non-selective admittance schools. 


Class Size 

Increasing class size is sometimes seen as a method to increase university revenues (Taft et al., 
2011). This can unwittingly place an increased burden on online faculty members who find that 
workload often increases when online class size swells (Taft et al., 2011). Increased class size 
decreases the amount of contact time per individual student, which causes faculty members to 
perceive a decline in the quality of the educational experience (Dykman & Davis, 2008; Taft et 
al., 2011). Taft et al. (2011) suggested that optimum online class size is somewhat determined by 
the mode of instruction and may be dependent upon where the instructional goals fall on the 
continuum of objectivist and constructivist theories. Also, these researchers stated that if a class 
falls along an objectivist pattern of teaching and learning, then the online class size can increase 
without detriment to the student educational experience. However, the more constructivist- 
based courses, which require increased instructor contact time, need to have lower enrollment 
to satisfy teacher workload and student satisfaction. They remarked that class size can also be 
thought of in terms of Bloom’s Taxonomy such that information on a lower level on Bloom’s scale 
may be taught and learned with a larger class size, while those classes that require students and 
faculty to work at higher levels of the Bloom Scale should enroll fewer students. Burruss, Billings, 
Brownrigg, Skiba, & Connors (2009) found that class size relates to certain educational practices 
and outcomes. For example, class size is not related to the use of technology or perceived 


satisfaction or professionalism; however, educational practices such as active participation in 
classes, interaction with peers, and student-teacher contact time were perceived as relevant to 
class size. Taft et al. (2011) found student satisfaction was negatively affected by increased class 
size in distance education. 


Due to the increased faculty workload, Tomei (2006) recommends that class size for online 
courses be kept to 12 students (compared to 17 students in a traditional format). However, Drago 
and Peltier (2004) found no relationship between class size and course effectiveness in their 
study range of 22-83 students. Orellana (2006) found that actual online class size was not related 
to an instructor’s perception of the course interactivity level nor actual interactivity level of the 
class, however instructors still perceived interactivity would improve with a smaller class size. In 
Orellana’s (2006) study, actual class size was 22.8 students, while the perceived optimal class 
size was 18.9 students. Maringe and Sing (2014) pointed out that there is no definitive definition 
of a large class, but that there is evidence (Cuseo, 2004) of diminishing returns in terms of 
educational effectiveness (opportunity to learn) as class size increases in traditional formats, 
such as with early undergraduate education. Lowenthal, Nyland, Jung, Dunlap, and Kepka (2019) 
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found that, in traditional online courses with large enrollment, students reported less satisfaction 
with the course and learning outcomes were significantly lower than in face-to-face formats. 


In the case of MOOCs, one confound to discussing class size is that many MOOC formats allow 
learners to begin the course at any time, even close to the time of the class closure (DeBoer, Ho, 
Stump, & Breslow, 2014; Leach & Hadi, 2017). In their study of xMOOCs, DeBoer et al. (2014) 
found that completing registration was the only contact one-third of enrolled students had with 
the course. They noted that, unless students wanted to opt-out of email contact, there would be 
no motivation to remove their names from the class list since there is no monetary exchange nor 
penalty for withdrawal. Further, DeBoer et al. (2014) remarked that class size versus completion 
rate in MOOCs is perhaps a reflection of class commitment that is in contrast to more traditional 
online offerings because students in a university setting share more commonality in their 
reasons for taking the online course and more shared learner characteristics. 


Course size in traditional or MOOC environments is not clearly defined or articulated in the 
current research. For the present, the Bloom’s taxonomy argument may provide guidance; that 
is, lower level information may be successfully taught and learned in a large class format, while 
subjects requiring higher order thinking would be best approached with a smaller class size 
(Taft et al., 2011). 


Blended Learning and Class Size 

Large lecture class instructors may perceive that blended learning is impossible due to class 
size since, in some cases, like flipped classrooms, the benefits of blending are derived from 
active learning, peer interaction, and other student-centered tools (Danker, 2015). Danker 
(2015) was able to use a flipped format using tools like peer learning, active learning, and 
inquiry-based learning and 90% of students reported engaging in connecting topics from 


previous learning. Deri, Mills, and McGregor (2018) found that structuring a previously small 
general chemistry class into a large (from 20 to 1000 students) class was possible using a flipped 
arrangement and demonstrated improved performance over traditional lecture-based learning. 


The improvement in performance was static across different instructors and different student 
demographics (Deri et al., 2018). Further, the benefit of the flipped arrangement benefitted 
students considered less well prepared for college (Deri et al., 2018). Similarly, Robert, Lewis, 
Oueini, and Mapugay (2016) found that using peer-led team learning allowed for content to 
remain consistent with traditional classroom instruction and attain higher achievement and 
higher knowledge retention than did traditional students. 


Brown, Karle, and Kelly (20158) found that studio learning could be achieved using blended 
methods when large classes (n =170) were subdivided into smaller sections (n =18) and further 
subdivided into teams of six students to give students stronger support. Brown et al. (2018) They 
demonstrated that the larger class size was no hindrance to achieving the practices and 
outcomes of more intimate studio courses when the blended design utilized collaborative 
technology platforms. Francis (2012) offered sage advice that using appropriate instructional 
strategies for blended learning, such as advance organizers, formative assessments during class 
meetings, class questions or polls, cooperative learning and reporting, exit tickets, minute 
papers, and encouraging student engagement in class activities instead of using class time for 
personal web surfing or social interactions. 


Trust 

According to Shaw (1997), trust is defined by three imperatives, namely results, integrity and 
concern. Results implies that people deliver on what was promised; integrity implies following a 
known paradigm of values, beliefs and practices, and concern is showing deference to the well- 
being of others. Trust in education can be considered in several contexts, including a student’s 
propensity to trust, communication, instructor characteristics and behaviors in online courses, 
organizational reputation, peer-peer relationships, policy structures, student control or 
empowerment, curriculum, and technology (Hai-Jew, 2006). Trust and privacy issues arise in e- 
learning in several activities, such as peer review, peer tutoring, learning object selection 
(reliability of the object or competence of the contributor), collaboration, group learning, role 
playing, evaluation, and personalization of the learning objects (Anwar & Greer, 2012). Studies 
outside education yield information that demonstrates that users of online health information 
systems place a high premium on trusting the ability and benevolence of the health infomediary 
(Song & Zehedi, 2007). Reputation, therefore, plays a significant role in a person’s ability to trust 
(Anwar & Greer, 2012; Song & Zehedi, 2007). 


Wang (2014) proposes a socio-technical framework to advance trust in online learning 
environments which differentiates several trust-inducing components into two categories. The 
two categories are course instruction and privacy and security. Course instruction includes prior 
positive online course experience and the good reputation of the online learning system or the 
instructor, design quality and high information level, contact details, instructor assertiveness, the 
responsiveness of the instructor, the sense of community and caring exhibited by the instructor, 
and reliable and timely course access (Wang, 2014). Privacy and security encompass the 
disclosure of appropriate security and privacy information, the use of system security measures, 
and third-party privacy and security features such as encryption (Wang, 2014). 


Trust from the Student Perspective 
Trust is a critical component of the online learning environment because of the nature of online 
interactions which can make participants vulnerable due to sharing stories and opinions with 
strangers (Hai-Jew, 2007). Trust in Western higher education is based upon multiple criteria 
such as instructors exercising appropriate boundaries toward students, respecting student 
privacy, respecting student differences, and not endangering a student’s free will. 


According to Hai-Jew (2007), instructors handle both truth and opinion while training learners to 
increase their learning or skills for future endeavors. For students to have high trust in their 
online instructors the teachers must engage in consistent and regular communication, be 
perceived as credible experts in their field, exercise sincerity, and be perceived as honest. 
Further, students expressed that instructors who showed personhood and engaged in personal 
sharing were trustworthy. Conversely, students can express a loss of trust, as early as the first 
log-in for the online course if the instructor has failed to put appropriate information and 
expectations in the course materials. Students felt their trust waver if the instructor gave out 
grades that were “unreal”. For example, if instructors gave too many high marks, were harsh or 
inconsistent in grading, or gave inconsistent feedback, students felt a loss of trust). Furthermore, 
some online technologies, such as those that use electronic surveillance technology to monitor 
student behavior in learning spaces, can provoke distrust. Wang (2014) found that there was no 
difference in trust producing factors between genders, educational levels, time spent in the 


course material, or previous online experience. Furthermore, students with disabilities reported 
that they would self-disclose their needs to an instructor if the instructor was deemed trustworthy 
by them, although 67% of the sample stated that they would only ask for accommodations if they 

felt that they needed them. 


Hai-Jew (2006), in research aimed at the creation of a survey instrument to measure student 
online trust (Online Trust Student Survey or OTSS), found that students naturally fell into “low 
trust” and “high trust” learner categories. Hai-Jew (2006) describes this phenomenon as trust 
propensity. Trust propensity is a person’s tendency to extend trust in the first place and it is 
linked to parental styles and attitudes (Hai-Jew, 2006), self-trust, and a person’s capacity to trust 
(Reina & Reina, 1999). 


Tmst from the Instructor Perspective 

In Hai-Jew’s (2007) study, instructors emphasized that trust was pivotal in both traditional and 
online course formats, and that the instructor trust paradigm began by trusting oneself to 
competently teach the material. Instructors noted that trust between the teacher and the student 
was maintained when instructors met stated expectations and when instructors were supportive 
of dissenting ideas and respected and welcomed the student participation, all of which resulted 
in greater expression by students. 


Instructors felt that trust in the online classrooms became evident by the third week and teachers 
felt an urgency to establish trust early by fostering rapport and relationships (Hai-Jew, 2007). 
This approach was echoed by Jaffe (1997) who encouraged quick response times and early 
student interaction. Instructors ranked peer-peer interactions as highly important and cited that 
peer trust was a result of respect for each other (Hai-Jew, 2007). Instructors encouraged student 
self-efficacy and warned students against excessive self-revelation in online environments. 
Instructors also warned that the student needed trust that the curriculum and materials would be 
relevant to future endeavors. 


Technology could also be a barrier to trust in the online environment, and instructors believed 
online educators needed to keep students informed through appropriate communication, such 
as alerting students to changes in the schedule, class announcements, or expectations, and 
modeling the traditional classroom experience (Hai-Jew, 2007; Wegner Holloway, & Garton, 
1999). Furthermore, educators encouraged other online instructors to verify that any student 
responses were free of innuendo or sarcasm (Hai-Jew, 2007). 


Trust from the Administrator Perspective 
In a study by Hai-Jew (2007), administrators reported that support for online education can take 
on several forms from mandating that faculty take training through a continuum of administrators 


keeping a hands-off approach to curriculum to maintain faculty freedom over their courses. The 
administrators polled all agreed that trust between the student and the instructor was important 
for the success of online courses, and that student trust could be undermined if faculty members 
did not meet student expectations. Additionally, faculty trust can be eroded when time spent on 
preventing cheating and focusing on determining if the class was successful take on a prominent 
role. In his study, administrators were noncommittal about the importance of peer-to-peer trust, 
although the consensus was that the importance of peer-to-peer trust in online settings was as 
important as in face-to-face courses. 


Administrators felt that having stable technology for the students to use in online courses was a 
central issue to successful online course development and that the development of contingency 
plans could be beneficial to deal with technology problems (Hai-Jew, 2007). Marek (2009) 
suggested that institutions provide strong technology infrastructure support for faculty with a 
centralized support area for online teaching and learning faculty, that teachers be given direct 
training opportunities to improve their technology skills, and that online instructors should be 
considered for incentives for online teaching in the form of retention, financial incentives, and 
tenure policies. 


Support services 

Lack of support can leave distance learning students feeling isolated and lacking in self- 
direction and management, which leads to withdrawing from the course (Ludwig-Hardman & 
Dunlap, 2003). Therefore, student support services are closely related to student retention. 
Support services begin with course selection and registration, including assisting students with 
financial aid (Lee, 2010). Student support services also assist students with all technology issues 
including browser compatibility and course access. Student support services fall into several 
general categories such as academic services (advising, library, financial, and admissions) and 
social services (student organizations, psychological services, placement services, and 
instructor support). The realm of assistance for students in online learning can encompass 
activities such as counseling or guidance, access to course materials and information, instructor 
feedback, computer services, administrative help, peer interactions, library access, and family 
support (Maritim & Getuno, 2018; Oothuizen, Loedolff, & Hamman, 2010). Service is an intangible 
attribute of care, and has difficulty being measured due to the subjective perceptions of the 
individual user (Duffy, 2008). Shea and Armitage (2002), in reporting on the Learning Anytime 
Anyplace Partnership funded by the U. S. Department of Education, suggested that student 
services are viewed as an ever-expanding web of interactions between administration, 
academic services, personal services, and communication services aimed at each individual in 
each course. 


Helgesen and Nessit (2007) found that student satisfaction was closely tied to student loyalty. 
Service quality is the driver of student satisfaction (Lee, 2010). Yukselturk and Yildirim (2008) 
found that students’ perceptions of classroom support can wane as the semester concludes, 
leading them to encourage course designers and instructors to be vigilant in keeping the 
community of online learners engaged through well-structured activities and interactions. Tait 
(2000) suggested that, due to the individual nature of each learning situation in regard to student 
factors (age, gender, academic level, income, language, special needs), geographic realities 
(physical and social), academic rigor of a program or course (teaching, delivery method, 
assessment), the technology infrastructure (personal and institutional access), the scale of the 
course (cost and flexibility), and the characteristics of management (quality assurance and 
learning management), there is no “one-size-fits-all” approach to providing student services. 
However, attention to general areas of support will help institutions make correct choices for 
their circumstances (Tait, 2000). 


Tait (2000) suggested viewing support through the triple lenses of cognitive, affective, and 
systemic support. Cognitive support for students encompasses the availability and suitability of 
course materials, while affective support calls for attention to the student environment such that 
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the student as self is nurtured. Systemic support involves the administrative policies and 
procedures that help students navigate the institution and the course. Some learning scientists 
believe that learner support should be the natural bridge from one activity zone to another, as 
posited by Vygotsky (Pelissier, 2019). In this model, online learning can provide the assistance 
to the learner that is needed for transitioning or expanding the zones for the student. 


Oothuizen et al. (2010) found that student satisfaction with student support services was low in 
the areas of university-provided counseling and advice, availability of learning materials, peer 
support, and administrative support. Administrative support scored the lowest in perceived 
support and there was a wide variance in support ratings in all areas, indicating that respondents 
were strongly critical of the support provided or strongly supportive of the support provided. 
Male respondents were more satisfied with institutional support compared with females, and 
overall regarded all areas of support services as less important. 


Tuquero & McCool. (2011) examined support services through a meta-ethnographic analysis of 
findings presented in multiple doctoral dissertations and found support for Floyd and Casey- 
Powell’s (2004) Inclusive Student Services Process Model. This model stresses that online 
learners want student support services, expect to receive them, and will evaluate a learning 
institution by their presence. Further, they stressed that students want support service from the 
point of entry into the institution (learner intake) until they complete their degree (learner 
transition). Students expect rapid response times for their inquiries and help through advising, 
career counseling, and library resources (learner intervention and support). The point of entry 
for the institution begins with student information on the college or university’s website. A study 
by Jones and Meyer (2012) found that student support information was evident for certain 
information, as follows: 


Online application to the distance program - 57% 

Online textbook ordering - 55% 

Online financial aid application - 52% 

Online registration - 50% 

Required technology, faculty contacts, and costs - 45% each 
e LMS training for students - 42% 


Additionally, Jones and Meyer (2012) found that the number of mouse clicks to get to many 
pieces of information necessary to students, could average over four clicks. These authors note 
that it often took an “extensive” amount of time to locate distance learning information and that 
students would not likely spend as much effort to locate services as the research team did. 


Resource deficits 

When considering online instruction, Simpson (2013) points out that e-learning can easily be 
confused with e-teaching. Simpson cautions that instructors should be careful not to put too many 
“e-teaching” devices onto learners, in hopes that meaningful learning will occur. Overload in 
decision-making situations can lead people to make choices based on emotion, rather than on 
best practices and can deplete mental resources (Schatz, 2019). Overload can create a deficit in 
attention and learner effectiveness. while thoughtful design can assist learner productivity as it 
eliminates the stress of too fast a pace and too many resources. 


To ensure that support measures are in place, policymakers must determine who will provide 
technology support services, and how those services will be offered (Giattino & Stafford, 2019). 
Policymakers will need to ensure that all stakeholders suffer no lack of resources to work in the 
online system, including money and manpower. In addition to money and manpower, Clapp et 
al. (2019) remind stakeholders that time for course development is also a resource to be 
considered. Library services are impacted in online courses and consideration of the institution’s 
library resources and availability for gathering course materials should be part of the 
development plan (Clapp et al., 2019). An intricate, fluid, and stable distance program will 
require that responsiveness be built into the online learning ecosystem (Giattino et al., 2019). 
Support measures must include the resources that support change in the learning ecosystems as 
distance learning programs are conceptualized and implemented such that an attitude of support 
underpins the cultural shift from an industrial model to an information model (Erb & Shaw, 2019). 
For more information on student and faculty perceptions on resource deficits, see “Student 
Barriers to Online Learning” in this document. 


Blended Learning and Trust 

Trust issues mentioned above would likely be operational in blended environments due to their 
online components; however, one aspect of trust that could potentially affect blended learning is 
a student’s distrust of modalities that are not textbook driven (Orton-Johnson, 2009). Orton- 
Johnson (2009) found that student non-use of blended content was due having trust in traditional 
texts as authentic academic knowledge. Students perceived that online materials were ‘non- 
academic’ and perceived there were more convenient, appropriate, and reliable sources of 
information than online content (Orton-Johnson, 2009). Students lacked self-trust and relied on 
items such as reading lists to anchor their learning, fearing that deviation would cause them to 
stray from the academically safe and know (Orton-Johnson, 2009). Fear of technology use was 
not seen as a contributing factor to students’ trust in online resources (Orton-Johnson, 2009). 


Technology considerations in online learning 
Technology integration 


One technology consideration in online learning is the need to assure students that they will 


have access to the newest technologies and that training in the new technologies is necessary for 
student success (Hafeez, Gujjar, & Noreen, 2014). Another goal is to equip higher education 
institutions with the ability to develop and maintain a flexible, technology-facilitated teaching 
and learning strategy (Lisewski, 2004). One caveat to that vision is the rapid change in 
technology, which in turn fuels pressures to implement the new applications quickly (Yohe, 
1996). Yohe (1996) bemoans that technology support services are tasked with not only 
delivering new technology for users, but also maintaining legacy systems beyond their 
reasonable life-spans, working to provide interoperability between applications that may be 
incompatible, and to do so with dwindling resources. He suggests that the challenge consists of: 


1) Providing sufficient connectivity through network design, robustness, and redundancy. 

2) Providing sufficient processing capacity through memory, speed, and bandwidth. 

3) Coordinating the service goals that consider the institution’s strategic plan concerning 
support priorities, guidelines for hardware and software roll-out and support, and 
decision-making roles of the support staff. 

4) Integrating new technology and bringing diverse technology solutions together. 

5) Maintaining aging technology and systems. 

6) Ensuring funding for technology infrastructure changes. 

1) Identifying and hiring qualified support staff. 

8) Delineating expectations for end-users. 


He asserts that planning and communication are fundamental qualities of an institutional 
technical support service. Unmet expectations can be alleviated through establishing advisory 
committees, stakeholder roundtable discussions, service agreements, and workload reduction 
policies with stakeholders. Further, he suggests making institution-wide communication 
appropriately when there are technology updates, problems, and fixes available. This can be 
accomplished through publishing contact information for a single point contact for users with 
questions or difficulties, automated problem-tracking, help-desk phone support, email, self-help 
conversion programs for user-encountered problems, hardware and software standardization, 
and frequently asked question pages designed to ameliorate slow response times for end-users . 


Support services and maintenance strategies 

Support services for technology can be contextualized by investigating the similarities and 
differences between staff or faculty members and students in their views and uses of technology 
(Waycott, Bennett, Kennedy, Dalgarno, & Gray, 2010). Prensky (2001a; 2001b) posited that since 
college students have been immersed in technology use for their entire lives, they think and 
process information in a different way than older generations. This potentially sets up a digital 
divide between the instructor and the student that would, if true, make communication 
challenging. Prensky (2001a; 2001b) calls the life-long technology users digital natives and the 
generations that have witnessed technology expansion, digital immigrants. This digital native’s 
fallacy has led organizations to think that students do not need technical support. However, this 
is not the case. 


Judd (2018) stated that while the concept of digital natives and digital immigrants resonates with 
educators, and while there is still continued interest in searches about these labels, there isa 
lack of scientific evidence to support the native/immigrant concept. Bennett, Maton, and Kervin 
(2008) stated that the digital natives debate has led educators to feel that they will be unable to 
teach the newest generations and calls the debate in academia a sort of ‘moral panic’. These 
researchers claimed that research fails to detect a generation that is identifiable as natives of 
technology nor even a technology user that can be distinguished as adept. The lack of support 
for the digital native/digital immigrant concept has been further substantiated by Waycott et al. 
(2010). They found that students used the same technologies as the faculty and staff and that it is 
used in the same type of contexts. For example, both groups tried to either adhere to personal 
guidelines for keeping technology for work and education versus for their personal lives 
separated or perceived it was fine to allow technology to merge their work/education and 
personal lives. They suggested that any differences in perspective may be due to the stage of 
life of the individual. One way in which students differed for faculty and staff were in the roles 


that technology played in academic endeavors. For example, students saw the benefits of 
technology were in its supporting role of fostering communication between students and staff 
and between students. Students also valued technology for its convenience in managing 
coursework. 


Three concerns voiced by students were not having appropriate access to technology, not 
understanding how to use certain technologies, and missing messages from faculty and staff 
(Waycott et al., 2010). They stated that these concerns may reflect that connectedness to 
technology in everyday life does not equate to using technology effectively in higher education. 
The Educause Center for Analysis and Research (ECAR), after analyzing responses from 183 
institutions worldwide, found that students felt strongly that technology enriched the learning 
experience, helped them complete the learning objectives of the course and was appropriate for 
the content (Brooks, 2016). Furthermore, two-thirds of students reported that their experience 
with campus WiFi was good to excellent, and about one-half of the students felt their instructors 
had the technology skills needed for course instruction and to connect to learning materials. 


Brooks (2016) also found that a student’s technology experience is influenced by their 
experiences with campus infrastructure and their attitude about the benefits of technology use in 
their future careers. Students who are female and first-generation college students were more 
likely to have efficacy, enrichment, and engagement levels raised by technology use. He also 
found that encouragement to use technology during class could provide a distraction for 
students; however, students who were acquainted with technology device usage before entering 
college were less distracted by technology in learning environments. According to ECAR, 98% 
of institutions in the U.S. provide online learning support for students. 


Staff and faculty see technology in education as a method for helping students to learn and to 
efficiently manage their instructional duties demonstrating that the same technology is used 
differently between staff or faculty members compared to students (Waycott et al., 2010). The 
ECAR study (Brooks, 2016) revealed that U.S. institutions offer a wide variety of services to 
instructors who are willing to incorporate technology into their learning environments. The types 
of opportunities available to faculty and students of U.S. higher education institutions include 
university-provided support through the instructional design process (89%), online learning 
support technology for faculty members (96%), faculty training in technology use (99%), an IT 
teaching center (79%), and faculty group training (98%) (Brooks, 2016). An early study showed 
that although faculty training is made available in many cases, attendance to training remained 
low (Flowers, 2000; Tuquero & McCool, 2011). 


Scalability 

Scalability in distance education is the ability to take smaller online course offerings and expand 
them to accommodate larger enrollments (Laws, Howell, & Lindsay, 2003). At the core of the idea 
of scalability is a prudent navigation of complex questions like: How do we attempt to expand the 
enrollment? and, if we do, what are the consequences to pedagogy, teacher workload, student 
outcomes, and financial status? . Technology is the backbone on which online education is 
carried so the institution’s capacity for technology-based learning must also be considered 
(Hossain et al., 2018). Obvious elements of the educational process, such as student skills 
assessment, must be moved into a large-enrollment context (Roberts, LoCasale-Crouch, Hamre, 


& Buckrop, 2017). Therefore, enrollment decisions become a balancing act to keep the distance 
education endeavor moving forward at a responsible pace (Laws et al., 2003). 


Without the cues and class interaction inherent in face-to-face courses, online instructors must 
build in appropriate student interactions prior to initiating the course, which requires that faculty 
not only be fluent in the technology of the online course, but also be adept at using the 
technology to engage students (Laws et al., 2003). Toward this goal, Hossain et al. (2018), in 
studies aimed at iteratively adapting a real laboratory (versus simulation) MOOC course to Next 
Generation Science Standards (NGSS), were able to utilize cloud computing to make an 
experimental lab course able to handle more than 30,000 experiments per week at low cost 
through software manipulation. This achievement represents the capability to adjust complex 
inquiry-based learning to large audiences. 


Laws et al. (2003) have posited that scalability, being multi-factorial, is best viewed as a 
continuum from marginally scalable to moderately scalable to highly scalable, based on the type 
of Bloom’s learning level the course is aimed toward, (remember, understand, apply, analyze, 
evaluate, create), the learner’s educational level, the retention level expected of the student, the 
program type and market (open versus degree; open-enrollment versus restricted enrollment), 
tuition burden, and instructor load, rank, and status 


Laws et al. (2003) proposed that instructors be allowed to personalize the amount of support they 
receive from the institution to assist them in scaling their courses for larger enrollments. Possible 
suggestions for assisting faculty in scaling their courses are: 1) a direct model of faculty-student 
interaction in which the online instructor maintains virtual office hours for students with the 
caveat that instructors set up response time and method criteria for students, 2) on-demand 
support in which a teaching assistant is used to help students with coursework questions or 
problems, 3) teaching assistant course development/mentorship, in which teaching assistants 
operate under direct faculty supervision to write or adjust course material for the faculty 
member and provide on-demand student or faculty support, and 4) student to student mentoring 
or service learning in which more experienced students assist newer students in their course 
completion . For more information about MOOCs specifically, see the summary of MOOC 
research in Appendix B. 


Governance of online courses 


General organizational considerations 

Including distance education in higher education institutions necessitates a balancing act 
between ensuring pedagogical effectiveness, understanding the learners, developing 
interactivity, designing strategies for student retention, negotiating faculty incentives, and 
evaluating profitability and affordability (Laws, Howell, & Lindsay, 2003). Distance education, in 
its infancy, started with a focus on the adult or life-long learner, as part of a college or 
university’s community offerings (Shattuck, 2014). However, online learning is growing, anda 
2013 study revealed that there were at least 6.7 million students taking at least one online course 
(Allen & Seaman, 2013). Taft et al. (2011) suggested that colleges and universities are scrambling 
to add online course offerings due to the rapid growth of technology, the change in students’ 
characteristics and lifestyles, a growing demand for educational access, and competitive forces 
within higher education. Distance education enrollment continues to increase at the same time 
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enrollment in higher education institutions is dropping (Lederman, 2018). In 2017, the number of 
students who exclusively study online rose to 15.4% while those that took at least some online 
courses rose to 17.64% (Lederman, 2018). At the same time overall higher education enrollment 
fell by 0.44% (Lederman, 2018). 


Many higher educational institutions struggle with sagging enrollment due to economic factors 
(Taft et al., 2011) and distance education can seem like a panacea to struggling universities that 
perceive increasing enrollment will help them improve revenues (Dykman & Davis, 2008). 
However, the quality of online courses is of concern to students, teachers, and administrators 
alike (Ossiannilsson, Williams, Camilleri, & Brown, 2015). Bailey, Vaduganathan, Henry, 
Laverdiere, and Pugliese (2018) stated that colleges and universities can develop online 
learning experiences that are high in quality if they choose to invest in strategic approaches. 
Wang (2014) stated that the United States has not developed nationally recognized, strong, and 
consistent quality-assurance measures for online institutions. Compora (2003) noted that many 
institutions do not have unique mission statements for online learning and that online courses 
may be implemented without the benefit of a needs assessment. It is established that thoughtful 
course design is capable of delivering equivalent or improved learning outcomes compared to 
traditional learning (Spiceland & Hawkins, 2002; Wegner, Holloway, & Garton, 1999), improved 
access for students, especially those who are disadvantaged, and improved institutional financial 
status due to reducing operating costs and growing revenue (Bailey et al., 2018). 


Management culture in organizations 


The growth of online learning necessitates the emergence of governing bodies within 
organizations to assure quality, maintain standards in technology usage, support stakeholders, 
and ensure accurate reporting of learning outcomes (Giattino & Stafford, 2019). Furthermore, 
online programs must incorporate plans for long-term sustainability (Angolia & Pagliari, 2016). 


Angolia and Pagliari (2016) asserted that to develop and sustain a quality distance learning 
program, the university must have sufficient infrastructures such as policies and processes, 
information and communication technologies, instructional support staff, technology hardware 
and facilities, and training. Furthermore, administrators serve to encourage trust, foster 


relationships, and find common ground for discussion and action between stakeholders, while 
collecting and using data to facilitate change and support faculty in the online education 
endeavor (Burnette, 2015). Each of these functions must be completed within the context of 
student satisfaction and retention, (Muljana & Luo, 2019) and with a constant growth and 
maintenance philosophy (Angolia & Pagliari, 2016). 


Further, administrators must have procedures in place that help instructors determine the 
student audience’s content readiness and skill level (Artz, 2011). Gaytan (2009) describes the 
overarching purpose of governance as a means of coordinating the plan, design, delivery, and 
assessment of online learning. Regardless of the individuality of each use case or learning 
endeavor, program administrators or overseers must answer the similar question of how they 
will centralize oversight of the learning ecosystem while remaining flexible to changing 
technology, tools, data collection, data usage, and stakeholder interests (Giattino & Stafford, 
2019). 


Administrators may have to overcome some personal bias regarding distance education and 
online learning as they approach the governance of this segment of their educational services. 
Burnette (2015), ina study of administrators, found that 66% of the respondents were “bound” 
with a traditional outlook toward distance learning that reflected a reticence to believe distance 
learning provides a quality education. Additionally, administrators may be caught in an authority 
struggle as distance education incorporates areas, like technology resources, for which they are 
not directly involved in decision-making. 


Giattino and Stafford (2019) suggested that there are four key areas of focus when formulating a 
governance plan for learning ecosystems. These key areas are membership, policy, processes, 
and resources. Due to the lack of face-to-face contact, problems that arise in online learning 
environments may escalate quickly, making both instructors and students lose enthusiasm for 
online opportunity (Dykman & Davis, 2008). Membership encompasses decision-making about 
who will be part of the learning ecosystem, how members will be able to interact with the 
ecosystem, and how decisions will be reached within the learning enterprise (Giattino & 
Stafford, 2019). Policy addresses who will make and enforce policy and how changes in the 
ecosystem will be implemented to the best advantage of the members. Processes that must be 
considered are how stakeholders will be using the system and how their creativity and 
experimentation within the system can be encouraged without risk to other members. Systems 
must be designed with consideration for how external partners can function within the 
ecosystem and how the system can remain relevant and responsive to the needs of the users. 
Determining who will be responsible for different resources in terms of manpower and funding 
is critical in keeping the ecosystem available and suitable for the different stakeholders. 
Equipment, technology support, and personnel support responsibilities must be assigned to 
maintain the learning ecosystem over time (Giattino & Stafford, 2019). 


Institutional support and compensation for online instructors 


To play a supporting role for faculty, administrators must first understand what motivates faculty 
members to undertake online course management (Parker, 2003). Faculty internal motivators 
are self-satisfaction, flexible hours, and the potential to reach a wider audience with their 
material. External motivators for faculty are monetary remuneration for teaching online courses, 
decreased workload, course development time, and technology for the faculty member’s 
personal use. Laws et al. (2003) found that faculty rank and advancement opportunities are 
motivating for faculty members’ willingness to participate in online course offerings. 


Marek (2009) stated that the increase in online courses offered at universities necessitates that 
instructors be grounded in sound pedagogical skills that can help make online courses 
successful and improve institutional quality. Compora (2003) noted that faculty may be selected 
to teach online courses based on willingness rather than ability (expertise). Faculty attitude and 
pedagogy are also cited by Angolia and Pagliari (2016) as being pivotal to online learning. They 
stated that distance education requires a faculty that shares best practices that are adaptable and 
adopt-able. Time commitments, student communication times and expectations, and constantly 
changing technology can tax faculty members. Lewis and Abdul-Hamid (2006) found that 
successful faculty members stress interactivity as key to successful online courses and that 
maintaining a high standard of course interactivity does not happen without intention. 


One barrier to effective online courses is that universities may lack a set of written guidelines for 
online courses and possess insufficient technical support for faculty members and students (Lion 
& Stark, 2010; Tallent-Runnels et al., 2006). Faculty members must have pedagogical support in 
using communication and technology tools (Angolia & Pagliari, 2016). Faculty must be trained in 
all new software that is onboarded because development and support is vital to a thriving 
distance education program. Ricci (2002) warns that an institution must have a comprehensive 
support structure in place for faculty, staff, and students with emphasis on technology support so 
that online courses are proficiently developed, and distance courses are not run in a continual 
crisis mode. 


In addition to pedagogical considerations, Tomei (2006) found that instructors in online courses 
can experience a fluctuation in the demands on their time due to an increase in instructional 
content hours, counseling hours, and assessment hours compared to traditional classroom 
formats. In Tomei’s study, content hours increased by nearly 18 hours while counseling 
increased by nearly 6 hours. The only construct that decreased teacher workload was 
assessment, which took about 4 hours off the instructor’s overall load. This decrease was being 
due to the formative evaluations being hosted, managed, and scored online. 


Bacow, Bowen, Guthrie, Lack, and Long (2012) summarized the perceptions of presidents and 
vice presidents (20 respondents), provosts (9 respondents), and deans, directors, and faculty 
members (14 respondents) in regard to online learning and course development from 25 
institutions that consisted of public, private, community colleges, and four-year institutions. 
Although the study was small, they elucidated key issues faced by the sample institutions. Key 
points from their interviews are summarized below. 


Key Stakeholder Observations Regarding Online Learning from Bacow et al., 2012 


Observation 


Online course approval 
undergoes the same process 


as traditional course approval 


No rigorous methodology is 
in place to evaluate learning 
outcomes in online courses 


Mixed reviews of student 
preferences of online versus 
traditional courses are 
confusing to interpret 


Mature, highly motivated 
learners disproportionately 
outperform other learners in 
online formats 


Cheating may be more 
difficult to control in online 
courses 


Student monitoring is easier 
with online formats 


Possible Result 


Faculty time constraints are 
overlooked 


Institutions may 
inappropriately rely on 
student retention data or on 
anecdotal evidence of 
learning outcomes (i.e., 
students in online courses did 
as well as traditional courses) 


When online and traditional 
formats are offered for the 
same courses, online sections 
fill up faster leading to 
assumptions that online is the 
preferred delivery method 


Student enrollment may be 
restricted to certain GPAs 
because online learning may 
seem too challenging for less 
academically sound students 


Faculty may alter course 
content to more project- 
based learning to curtail 
cheating 


Instructors can identify 
students that are not 
interacting with material 


Strategies for Overcoming 
Barriers 


Provide generous technical 
support for instructors 


Provide faculty support for 
online instructors 


Recognize innovative online 
teaching 


Identify courses that can be 
converted to online formats 
easily as the university 
begins online instruction 


Separate online 
administration from the 
traditional administrative 
entity 


Share revenue generated 
with departments using 
online learning formats 


Some faculty resist online 
formats because it runs 
contrary to their professional 
goals of student interaction 


Instructors may feel online 
learning will lower faculty 
employment levels 


Instructors must have 
technical aptitude and must 
re-develop courses for an 
online format 


Instructors are reluctant to 
teach courses that are 
developed by third party 
vendors or to “re-purpose” 
older material 


Uncertain intellectual 
property rights exist 


Instructors may gravitate 
toward teaching how they 
were taught which could 
diminish their desire for 
teaching online 


Administrators must 
counteract faculty, parent, 
and student perceptions that 
faculty/student ratios will 
increase 


Instructors may have 
increased time commitments 
in transitioning to an online 
environment 


Administrators must develop 
course “ownership” policies 
to counteract the perceived 
loss of instructor control or 
loss of customization 


Course content ownership 
must be addressed 


The table above reflects the concerns of higher education instructors who are developing 
content and teaching online. The Massive Open Online Course or MOOC has unique 
considerations beyond those listed in the chart above. For example, MacLeod, Haywood, 
Woodgate, and Sinclair (2016) remind the educational community that, while instructors are 
supposed to design their curriculum with the learner in mind, ina MOOC setting, that goal 
would be nearly unachievable. Additionally, the call for a constructivist community causes 
instructors difficulty in course design and delivery in the MOOC format due to the possibly 
massive scale in the number of participants. The constructivist agenda would urge instructors to 
foster teacher presence for improved learning (See Community of Inquiry in Section 5) 
(Macleod et al., 2106). Furthermore, educators are faced with the possibility that the 
demographics of the current group of MOOC participants may not mimic the next group such 
that the designs and delivery are not reusable without modification. It is also necessary for 


instructors to account for the multinationality of the MOOC learners, which could further burden 
the instructor’s design and delivery options (MacLeod, Haywood, Woodgate, & Alkhatnai, 2015). 


To counteract faculty reluctance, administrators must include faculty in discussions of an 
institution’s position, goals, and plans for web-based learning (Lion & Stark., 2010). McQuiggan 
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(2007) uncovered that there are four key elements to include in faculty development to prepare 
instructors for the challenges of online teaching. These include preparing course materials, 
learning to navigate the unfamiliar online environment, developing a thorough faculty 
development program, and using adult learning theory in faculty development programs. 


Martins and Nunes (2016) found that faculty undertaking online courses perceive that they 
reshuffle time for course development and preparation by increasing the duration of time spent 
in these activities. Instructors felt that teaching and learning activities were increasingly taking 
place over extended time periods and that this set up a competition for the academic’s attention. 
Instructors are required to quickly adapt to changing course delivery methods while maintaining 
their other career and professional performance requirements which are likely to esteem 
research over teaching and undervalue ‘teaching hours’. Additionally, the online course 
delivery can extend teachers in other ways, such as dealing with the communication difficulties 
that must be overcome in a community of inquiry situation that exists between instructors and 
students if the instructor is to maintain student engagement. Academics must also deal with the 
complexities of designing online courses with the expertise of a learning scientist so that 
students can maximize learning outcomes. Online instructors trying to promote effective online 
learning must be able to undertake technical challenges and expectations while trying to pace 
content and the temporal challenges of providing content. They stated that these temporal 
changes result in an academic’s perception that workload metrics, patterns, and conflicts are 
disrupted and that entrenched organizational policies fail to help instructors overcome or 
modulate these competing elements. These researchers called for guidelines that would 
establish new and reasonable norms regarding an academic’s virtual presence that would 
account for workload allocation frameworks and would implicitly address rewards for time in 


performance appraisals that are related to the scholarship of online teaching. Secondly, they 
stated that guidelines should be developed that address expectations of an online instructor that 
would be communicated to students about rules of conduct within an online course and 
instructor availability. 


Lawler and King (2001) proposed a model for faculty development called “The Adult Learning 
Model of Faculty Development” which suggests a framework of administration and faculty 
cooperation to pre-determine what will be covered in the faculty development, how it will be 
structured, and how follow-up on the training sessions will be completed. The steps of this form 
of faculty development include pre-planning for the development opportunity which addresses 
the culture and mission of the faculty opportunity (Lawler, 2003). A second planning stage 
addresses how the activities for the faculty development will be designed and implemented. A 
third stage monitors the delivery of the information to make sure that the participants are gaining 
the necessary opportunities for learning and are being instructed with adult learning methods in 
mind. Finally, a follow-up of the learning event allows participants to be supported in using their 
new knowledge. All elements from pre-planning through follow-up are conducted in a climate 
respectful of the way adults learn (Lawler & King, 2001). In online learning, Taylor and 
McQuiggan (2008) found that faculty were interested in learning about and having access to 
design strategies and use tools to support online learning and having information on how to best 
structure courses for an online environment. While faculty were interested in technology-skill 
learning, they were also interested in the implications of their instructional designs for effective 
online learning (Taylor et al., 2008). McQuiggan (2007) notes that faculty need encouragement to 
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reflect on their instructional practices to help them, as the learners, to make critical 
transformation in their thinking and behavior. 


Fair evaluation 

DeCosta, Bergquist, Holbeck, and Greenberger (2016) and Berk (2013) lament the lack of 
research into faculty evaluation for online teaching despite the wealth of research into faculty 
evaluation in traditional settings. Online teaching effectiveness is linked to more than effective 
pedagogies and instructional techniques that set the tone for the class; teaching largely depends 
upon the beliefs and attitudes of the teacher (Welch, Orso, Doolittle, & Areepattamannil, 2015). 
Furthermore, students’ expectations may cloud the definition of an effective online experience. 
Compora (2003) discovered that there is a general trend for institutions to conduct course 
evaluations, however there is little consistency in the method or requirements in evaluations. 
DeCosta et al. (2016) found that online instructors desire more holistic evaluations from multiple 
stakeholders especially regarding content, and that instructors desire evaluations so that they 
can become better teachers, not just better online teachers. Course format was found to have 
little interplay in student course evaluations between blended, online, and face-to-face 
environments leading researchers to state that the lines between student perceptions of delivery 
are diminishing (Dziuban & Moskel, 2011). 


Instructor evaluations 

Berk (2013) identified several guidelines that are currently in use to evaluate online course 
effectiveness and inform decision-makers. These include instructor-developed scales, which 
place the sole responsibility for the evaluation on the instructor and have the disadvantage that 
some instructors may not have any training or skill in developing surveys or evaluations. Some 
institutions rely on traditional evaluations used in face-to-face courses, which neglect the 
uniqueness of online instructional methods. Other institutions revise the face-to-face scale or 
supplement the face-to-face scale with additional items pertaining to online aspects of the 
course, such as technology. The addition of questions to a face-to-face survey can help in 
comparisons of the two learning modalities, if constructed correctly (Berk, 2013). 


Another rating method is to develop a completely new survey for use in evaluating online 
courses, however this may be cost-prohibitive and unnecessary, as some questions used ina 
face-to-face survey could be useful in evaluating online courses (Berk, 2013). A final method of 
course evaluation currently in use is to take advantage of commercially available or previously 
published rating scales Berk warns that validity and reliability of the commercial scales have not 
been reported; however, several published scales, such as the Students’ Perceptions of Online 
Courses (SPOC) scale or the Student Evaluation of Web Based Instruction (SEWBI) scale are 
available for use. A third published scale is the Student Evaluation of Online Teaching 
Effectiveness (SEOTE) scale. 


Student Evaluations 

Student evaluation is impacted by the online learning modality in that peer- and self-assessment 
have been introduced into the learning environment as both a learning activity and assessment 
tool (Dominguez, Jaime, Sanchez, Blanco, & Heras, 2016). System-derived learning analytics can 
help stakeholders understand the quality parameters in online courses (Perkins, 2019). In fact, 


Perkins (2019) admonished instructors to design evaluation points throughout the online course 
cycle to make on-going changes to best reach students’ expectations and improve learning. 


Reliability of the evaluations can blur the effective utilization of student reviews. Some confounds 
to evaluating student online learning have been reported, such as instructor accent (Sanchez & 
Khan, 2016). Sanchez & Kahn (2016) demonstrated that while perceived fluency did not alter 
actual learning, instructors with accents were rated as less effective. Carpenter, Wilford, Kornell, 
and Mullaney (2013) showed that “fluent” video instructors (those who stood up, made eye- 
contact, and spoke fluently) outranked video instructors who were disfluent (those who slumped, 
looked away, and spoke haltingly) in effectiveness. Student perceived learning was significantly 
higher when the instructor was fluent versus disfluent, although actual learning between 
instructors was equivalent (Carpenter et al., 2013). In learning environments utilizing self- and 
peer-assessments, Dominguez et al. (2016) found that self-reported assessments tended to be 
inflated and peer-reported assessments were subject to a friendship bias. In the same study, 
Dominguez et al. (2016) found that student-evaluators operated with a competitive bias, as well, 
when grading students from another institution in cross-institutional collaborative activities. 
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Appendix B: Courseware & Distributed Technology Review 


Technology is the fastest growing element of the modern learning ecosystem. Possibly 
because of this new factor, there are many common beliefs that learning with technology is not 
as effective as traditional classroom-based learning. This is a common belief among 
administrators and instructors which has served to hinder online and technology-led learning 
(Roby et al., 2013). However, under the correct circumstances, learning with intelligent 
systems (Craig, Hu, Graesser, Bargagliotti, Sterbinsky, Cheney, & Okwunabua, 2013; 
VanLehn, 2011), eLearning (Bernard et al., 2011), blended learning (Liu et al., 2016), and 
learning at scale (Taft et al., 2011) can be just as effective as standard classroom learning. 


State-of-the-art distributed learning environments should be supported and evaluated 
by data. 

State-of-the-art distributed learning environments should use video to present 
procedural interactions and model behavior not as a lecture replacement. 
State-of-the-art distributed learning environments should use virtual reality, 
augmented reality, and Simulations when there is a reusable topic that requires a 
setting that is interactive and requires real time human collaboration. 

State-of-the-art distributed learning environments use Social Media improve to 
interactions and engagement for online learning especially as class size increase. 
State-of-the-art distributed learning environments can use microlearning principles to 
support mobile learning within existing classrooms 

State-of-the-art distributed learning environments use UX/human-centered evaluation 
to increase understanding for all aspects of the learning system. 


Using Data in Distributed Learning Environments 

Before moving into specific types of courseware, it is important to understand how courseware 
components can interact with one another. We begin with discussions of Sharable Content 
Object Reference Model (SCORM) and Experience API (xAPI). 


Learming management systems/xAPI/SCORM 

SCORM is not in itself a specification or a standard (Bailey, 2005). SCORM describes how 
learning content is presented to the learner through a virtual learning environment (VLE). Also, 
it describes how the learner’s progress is tracked by the VLE (Bailey, 2005). SCORM enables 
learning content authored by a vendor to be easily imported and run in any SCORM conformant 
VLE. Assets (e.g., text, images, sound, assessment) or files that could be rendered by a web 
browser are assembled into a Shareable Content Object (SCO). These are then described 
through the addition of meta-data. A file called a package manifest is then created to package 
SCOs into a course structure. Usually, it includes a table of contents to enable learners to 
navigate between SCOs. This is exposed in the VLE user interface. 


How was SCORM used? 

SCORM has been used in learning management systems (LMSs) (Watson & Hardaker, 2005) and 
e-learning courses (Savic & Konjovic, 2009) to provide students the appropriate resources based 
on their learning styles. Watson and Hardaker (2005) discussed how LMS are extendable to 
provide guidance to learners by using SCOs. In their study, courses were represented as small 
discrete reusable 5COs. Their strategy was not considered dictatorial as it provided guidance in 
the form of different routes through SCOs to meet the same learning objective. Students were 
asked to answer a set of questionnaires to determine their learning styles. Using this information, 
a predefined manifest file of SCORM was then used based on the learning style of the student. 
This “personalizes” the order of the resources in the LMS. In another study, Savic and Konjovic 
(2009) designed a personalized e-learning course based on the student’s learning style using 
SCORM. This module was then integrated into the Sakai system. The learning style was based on 
the Felder-Silverman model and students were asked to fill in a questionnaire to determine their 
learning style. 


What is xAPI? 

Experience API (xAPI), also known as Tin Can API, was developed by the Advanced Distributed 
Learning (ADL) Initiative as a standard for describing learning activities that can be shared 
across systems. xAPI is one of many components of the next generation of SCORM (Poltrack et 
al., 2012). xAPI was conceived by applying the concept of activity streams to e-learning 
(Cooper, 2014). Events are captured as statements, which consists of three parts, namely an 
actor, a verb, and an object. Contextual information can also be added to provide more details 


on the learning activity. The granularity of information relies on the decision of the activity 
provider. Statements are built using Extensible Markup Language (XML). xAPI is typically used 
along with a Learning Record Store (LRS) which stores these activity statements. xAPI also allows 


for the retrieval of these statements, can exist on its own or within a Learning Management 
System, and because of the ability to be able to collect learning data across multiple systems, it 
has the potential for personalization (Durlach, Washburn, Regan, & Oviedo, 2015). 


Murphy, Hannigan, Hruska, Meford, & Diaz (2016) stated that although several standards are 
already in use in the context of training technology (e.g., high-level architecture and distributed 
interactive simulation), only xAPI that is able to capture and share human performance data 
because of how it was designed. All learning experiences can be represented, even those done 
outside of the training environment. 


Evidence for xAPI 

Long et al. (2015) developed an interoperable performance data for unstabilized gunnery 
simulators. The goal was to improve the efficiency of the adaptive training curriculum on a virtual 
simulation training system. They found a significant reduction in the amount of time to train with 
comparable final qualification scores. The Army Research Laboratory developed Pipeline which 
is a Microsoft.NET dynamic link library that enables simulator vendors to wrap around their 
systems to be able to generate and consume xAPI activity statements (Long et al., 2015). Like the 
result found by Murphy et al. (2016), a nearly 40% reduction in time spent training on Basic Rifle 
Marksmanship was found. This was mainly due to acceleration in the curriculum. However, in 
this study the participants were cadets from a local ROTC and not actual military trainees. 
Furthermore, both studies addressed only a stove-piped learning episode (i.e., across multiple 
learning episodes), as both implemented adaptation in a single learning experience (Smith, 
Gallagher, Schatz, & Vogel-Walcutt, 2018). Smith et al. (2018) stated that ideally, these 
adaptations should be applied within and across learning and development episodes. 


Uses of xAPI 


Several case studies have been done to explore the possibility of using xAPI to improve student 
learning. McGaghie, Issenberg, Petrusa, and Scalese (2010) noted that embedding xAPI in 
simulation-based team training could provide the potential to close the gap between simulation 
and real-world medical practice. This is facilitated by the ability to collect objective and detailed 
data which closes the gap between task performance and immediate feedback of that data. 
McGaghie et al. (2010) successfully demonstrated that xAPI could be a useful tool for collecting 
and visualizing data from multiple sources in relation to the Internet of Things. The context of 
their proof of concept is in a medical team training and simulation process. The goal was to 
improve training operations in environments and contexts that are high-stress operating 
environments. 


Scharlat (2013) explored using xAPI to collect data to devise a personalized, avatar-based virtual 
advisor using the vast data collected about the user (e.g., learner’s likes and dislikes). These 
virtual advisors are to be hosted in immersive virtual environments (IVEs). One of the potential 
functions includes highlighting new items in an annual training module. Throughout the module, 
the learner could focus on the new content. It could also have the potential to consider the human 
aspects of the learner (e.g., represent itself as a male since it has a lower frequency voice or 
make assumptions, such as the learner has a slight hearing loss). Scharlat’s work is an initial step 
in identifying useful data that could be used to create a personalized learning experience. 


One proposed application of xAPI is in tracking activities in transmedia training environments, 
which is a transformational technique applicable to training and education (Raybourn, 2014). Lim 
(2015) discussed two case studies at a high-level. The first one is an educational game Oregon 
Trail where xAPI was used to track activities that are effective in helping people learn. Since the 
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source code of the game is not readily available, a simple program was developed to record 
xAPI statements directly into the LRS. The second case study evaluated the proposal of the 
Learning, Interaction, Mentoring, and Evaluation (LIME) model to develop a recommender 
engine for students (Corbi & Burgos Solans, 2014). The LIME model could potentially capture 
both the formal and informal learning processes. This model captures four separate pedagogical 
components that are evident in all stages of education. This is a rule-based recommendation 
model that requires inputs that can be obtained in various ways. Furthermore, Lim (2015) 
highlighted the necessary adaptations and modifications for xAPI sentences to build LIME- 
compatible inputs. 


Durlach et al. (2015) discussed a proof-of-principle project where they used xAPI on an 
instrumented rifle range. The goals were to collect essential information training data to be able 
to: support individual feedback, aggregated data views for trainers and range operations 
personnel, flexible data views for training researchers, and automated availability of 
qualification data to the Army Training Management System (Durlach et al., 2015). xAPI was used 
to collect learner data and make it available across different types of learning systems. The goal 
is to support personalized education and training, as well as provide detailed feedback since the 
current system only provides trainees with a composite score regarding their performance. The 
study by Durlach et al. (2015), discussed the conversion of government-provided data from the 
LOMAH-TRACR (Location of Miss and Hit-Targetry Range Automated Control and Recording) 
system into xAPI statements. One integration issue was associating a soldier’s Electronic Data 
Interchange Personal Information (EDI-PI) with an individual’s data. This required the ability to 
"login" to identify the soldier. However, in training simulations, these are non-existent. 


Goodwin, Murphy, and Medford (2016) leveraged xAPI to produce human performance data that 
has intersystem data value. They developed a library of measures and an xAPI registry to 
encode this library of measures. They proposed a system called Support for Training 
Effectiveness Analysis with Data Interoperability (STEADI) which is an effort to develop an 
integrated performance measurement system to support the Integrated Model of Training 
Evaluation and Effectiveness (IMTEE) in a marksmanship use case. It serves as a stand-alone 
reference for training vendors to easily incorporate marksmanship-specific xAPI measures. One 
strength was that it did not prescribe how a performance measure was to be described. 
However, one weakness was due to its flexibility, as it may be prone to redundancy 


Sottilare, Long, and Goldberg (2017) talked about how to enhance the xAPI to improve domain 
competency modeling for adaptive instruction. This work in progress attempts to improve xAPI 
statements by documenting the following: achievement types, experience duration, experience 
source information, domain learning, forgetting, and assessment within learning experiences. 
These were identified in the literature associated with domain competence. 


It has been claimed that 70% of the learning activities are informal (Bogan, Bybee, and 
O’Connell, 2018; McCall, 1985), while that number is likely an overestimation (See Clardy, 
2018), Bogan Bybee, and O’Connell (2018) are likely accurate in their assertion that these occur 
outside digital learning environments (Bogan et al., 2018). This leads to a skewed interpretation 
of the effectiveness of training within these areas. In their study, Bogan et al. (2018) highlight the 
importance of capturing non-digital learning events. They proposed a strategy of using xAPI 
profiles to supplement metadata. As a result, efficiency was improved along with the user’s 


learning engagement. Furthermore, in this proof of concept it was found that the test group hada 
shorter test and virtual simulation duration and had improved test scores compared to the 
control group. Bogan et al. (2018) acknowledged that this study had a small sample size and so 
the results should not be viewed as conclusive. 


xAPI has also been proposed for use in standardizing self-regulated learning (SRL) traces 
(Manso-Vasquez, Caeiro-Rodriguez, & Llamas-Nistal, 2018). This would enable data collection 
from multiple sources and could be achieved using predefined recipes that could be used to 
monitor self-regulation. Similarly, it may sometimes be possible to extract xAPI statements even 
when the courseware is non-compliant. For example, Presnall and Radivojevic (2018) performed 
a case study of the implementation of xAPI in a Computer-Assisted Exercise (CAX), Viking 18. 
They implemented xAPI across several e-learning courses. They were able to extract xAPI 
statements from various non-compliant coursewares. This proof of concept in the context of 
training allowed them to perform learning analytics at a large scale and enabled visualizing 
disparate types of data in real-time. 


Future Research 


Our review identified several promising future research directions. For instance, Sottilare et al. 
(2017) hoped to expand the diversity of training domains to which xAPI statements are being 
used. Johnson, Nye, Zapata-Rivera, & Hu, X (2017) acknowledged two trending areas in the 
learning technology: Intelligent Tutoring Systems (ITSs) and increased data interoperability. 
There is also a possibility of implementing xAPI within or across ITSs (Johnson et al., 2017). 


Another area of potential future research is regarding system adaptation. Adaptation should be 
done at the system level where it is not a stove-piped approach (Smith et al, 2018). This will 
optimize the system and provide a better adaptation to the needs of the learner and at the right 
time. Finally, there is ongoing research on xAPI as one of the standards in the specifications of 
the Spiral-2 of the Total Learning Architecture (TLA). The vision is for an interconnected learning 
“ecosystem” to optimize talent management (Smith et al., 2018). 


Learning analytics 
The Goals of Learning Analytics 


Learning analytics (LA) is a fast-growing area of technology learning research (Ferguson, 2012). 
The first Learning Analytics and Knowledge (LAK) conference, defined LA as “the measurement, 
collection, analysis, and reporting of data about learners and their contexts, for the purpose of 
understanding and optimizing learning and the environments in which it occurs” (1st 
International Conference on Learning Analytics and Knowledge, 2010, para. 5). LA is an 
educational approach that is guided by pedagogy (Greller & Drachsler, 2012). LA aims to exploit 
the potential offered by the explosion of big data (i.e., interaction data, personal and academic 
information) which was brought about by the widespread use of online learning environments 
(Ferguson, 2012). One of its major concerns is to build trust and confidence in learning analytic 
tools. 


Ferguson (2012) identified the different factors that drive learning analytics. This includes big 
data, online learning, and politics. Some important motivations for learning analytics research 
are increasing motivation, autonomy, effectiveness and efficiency of learners and teachers 


(Buckingham Shum, Gasevic, & Ferguson, 2012). Papamitsiou and Economides (2014), in their 
systematic review of learning analytics research, found that most of the learning settings that 
would benefit from LA were virtual learning environments (VLE) or learning management 
systems (LMS), cognitive tutors, class-based and web-based environments, mobile settings, and 
recently, massive open online courses (MOOCs) and social learning platforms. Additionally, 
Papamitsiou & Economides (2014) identified classification as the most popular LA method for 
analyzing the collected data and found that clustering and regression (both logistic and multiple) 
have been used in data analysis. They also noted that recently, there has been popularity 
gaining on the use of discovery with models. 


Papamitsiou and Economides (2014) identified that student/student behavior modeling, which is 
focused on detecting, identifying, and modeling student learning behavior, was a prominent 
research purpose, the goal of which was to identify student learning strategies and 
circumstances under which the strategies occur, by modelling affective and metacognitive 
states. There has been an interest in the discovery and modeling of student behaviors within 
MOOCs, as well. The interpretability of the models still depends on the human and may not be 
interpreted the same among teachers. Another research objective is that of identifying, 
exploring, and evaluating student performance factors for the purpose of predicting student 
performance. These factors typically include, but are not limited to, grades, engagement, and 
demographics. Papamitsiou and Economides (2014) noted that more variables do not 
necessarily improve prediction accuracy in mathematical models. However, better results were 
found when using neural networks method compared to the regression analysis method. 


Another valuable use of LA is to increase student self-reflection and self-awareness. This 


increase is achieved by informing instructors of “disconnected” students. Students can also be 
evaluated through a student’s visualization feature, which informs learners of their performance 
and their personal progress as well as their performance compared to peers, usually in the form 
of a dashboard. 


LA can assist stakeholders by providing a method to predict dropout and retention. This is one of 
the key issues in LA/EDM research. By focusing on using data captured (sometimes early data 
about the students such as entrance exam results), these tools can alert instructors of the need to 
provide intervention to students. Observing students’ interaction with the system can be 
valuable as well. It was found that a combination of machine learning techniques yielded a 
higher accuracy, which depended on the granularity of student data. In fact, most of the learning 
analytics systems are capable of reporting interaction data of students to instructors or 
administrators (Schwendimann et al., 2017). Another use of LA is to improve feedback and 
assessment services where the goal is to provide meaningful feedback. This feedback can be in 
the form of adaptive assessment or formative assessment. Finally, LA can assist with the 
recommendation of resources, which could be recommended based on the affective state of the 
learner, using collaborative filtering, or through a hybridization between learner and content 
modeling. 


Multimodal Learning Analytics (MMLA) 


Blikstein (2013) argued that new insights into student’s learning trajectories could be provided 
by multimodal learning analytics, especially with the growing number of technologies that 
collect student artifacts. This is the combination of collection and analysis which could provide a 
novel approach to understanding when students generate solutions to problems or collaborate 
with peers, both in the digital and physical worlds. 


Another goal is to extend the application of tools and methodologies of LA to learning contexts 
where digital traces are not readily available (Ochoa, 2017). Multimodal interaction makes it 
possible to track multiple human activities, such as wearable cameras, biosensors, and eye 
trackers, which can be integrated to evaluate complex cognitive abilities. Learning aspects 
addressed in the current research includes the following: lectures, oral presentations, problem- 
solving, construction exercises, and the use of intelligent tutoring systems (ITS). Additionally, 
Ochoa (2017) asserted that the objective of MMLA is to combine these multiple sources or traces 
into a single analysis. The following are some of the various modalities used in MMLA research 
which are relevant for learning, but it should be noted that this is not a comprehensive list: gaze, 
body language (posture, gestures, and motion), actions, facial expressions, speech, and writing 
and sketching. 


However, there are some issues to consider when implementing MMLA (Ochoa, 2017). One 
concern is associated with recording data for a specific modality. This involves the acquisition, 
installation, and use of equipment. Another issue is concern about the privacy of the participants. 
Equipment used to capture data include cameras, microphones, and sensors. The next issue is 
concerned with the integration of multiple data coming from multiple sources. The variability 
can range from the extraction process, granularity, and format would have to be considered. 
This highlights the importance of coming up with a general framework that could guide the 
general LA community. The last issue is concerned with the impact on learning. With the 
complexity of the data acquisition and analysis involved, as opposed to a monomodal analysis, 
the positive impact on learning can be large enough to compensate for this complex approach. 


Leveraging Learning Analytics 


Arnold and Pistilli (2012) developed a system called Course Signals. This system leveraged the 
power of learning analytics to allow instructors to provide real-time feedback to students 
through faculty dashboards. Predictive models were run upon the request of instructors. This 
used the vast student data that is captured by multiple systems in the university. The goal is to 
identify students who are at risk and produce “actionable intelligence.” This intelligence could 
be used to guide students to appropriate help resources along with an explanation on how to use 
the resources. The Course Signals system has four components: performance, which is the 
percentage of points earned in the course to date; effort, which is based on how the student 
interacted with the university’s learning management system in comparison with his or her 
peers; prior academic history, which is comprised of high school GPA and standardized test 
scores; and student characteristics, which includes age, residency, and credits attempted. 
Students were not placed at risk simply due to a single factor, rather a risk was determined 
based on the student’s contextual landscape which converts both static and dynamic data points 
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into a single score that were used for the prediction. With this system, the researchers found 
significantly higher retention for students who have used Course Signals at least once (96.71%), 
compared to those who have not used at all (83.44%). Furthermore, students who had used 
Course Signals more than once had a higher retention rate than those who used it only once. 


Learning analytics have also been used to improve teamwork assessment (Fidalgo-Blanco, Sein- 
Echaluce, Garcia-Penalvo, & Conde, 2015). These researchers proposed a learning analytics 
system which aims to reduce the time spent for individual assessment in a teamwork assessment. 
Teamwork competency cannot be assessed only based on the group's results but should also 
evaluate the activity of everyone. However, this is tedious and time-consuming, especially with 
the voluminous amount of data being produced by education systems. There is a need to assess 
the real evidence of the work of each team member. This is achievable through studying the 
interaction (both active and passive) between students which could be used to infer individual 
performance of the teamwork context. Fidalgo-Blanco et al. (2015) used interactions within 
forums in Moodle to provide the teacher with the monitoring and evaluation data of individual 
members in the team. This system afforded real time extraction which promoted informed 
decisions. 


Future Directions of IA 


Ferguson (2012) identified four of the future challenges in learning analytics: 1) To build strong 
connections between LA and the learning sciences as work that focuses on cognition, 
metacognition, and pedagogy is under-represented in most of the key references. 2) For 
research to develop methods of working with a wide range of datasets which allows for the 
optimization of learning environments. To achieve this, complex datasets such as those outside 
the formal learning environments (e.g., biometric data, mobile data, mood data) must be 
factored into the analysis. 3) There must be a focus on the perspective of learners which would 
address their needs. 4) A clear set of ethical guidelines that must be developed and applied. 
Currently, there are no clear guidelines regarding the rights of learners in relation to their data 
or even their responsibility to act on any recommendations provided by learning analytics. 


Papamitsiou & Economides (2014) also analyzed the strengths, weaknesses, opportunities, and 
threats of the LA/EDM research by looking into selected case studies. One of the strengths was 
the availability of big educational data, while one weakness was the heterogeneous nature of the 
data sources which could lead to data representation issues. In addition, one opportunity was the 
exploration of the roles of self-reflection, self-awareness, and self-learning in intelligent, 
autonomous, and massive systems, however a threat was the issue of data privacy and ethics. 


Similarly, Campbell, Deblois, and Oblinger (2007) raised some of the issues that must be taken 
into consideration with the growing popularity of learning analytics. These issues are listed in 
the following table and are briefly described. 


Issue 


Big Brother 


Holistic View 


Faculty Involvement 


Profiling 


Data Privacy 


Data Stewardship 


Information Sharing 


Obligation to Act 


Distribution of Resources 


Brief Description 


Who determines what is going to be tracked as the notion of being 
tracked is threatening to some; is there a way to opt-out? Should we 
inform them? Do we need to ask for their consent? 


The prediction may only capture a certain aspect of the learner and 
may not provide a complete picture. Some aspects may not have 
been captured or explored. 


Faculty play a crucial role in the intervention process that 
addresses students who are at-risk. 


Allows for the creation of profiles of successful and unsuccessful 
students which could be used for potential interventions (e.g., 
prompts) or predictions. 


Who may be able to access the data as certain privacy regulations 
may protect these data about students (i.e., FERPA)? 


Data may be coming from multiple systems that could eventually 
be housed in a data warehouse. Also, how will these data be 
preserved, secured, and shared? Who may access this data and 
who can make decisions over them? 


To whom can we share models of successful students? Should these 
be shared with students, faculty, or other staff? 


As such student models provide a probability of student’s success, 
are the students, faculty, or the institution obliged to act upon 
these? 


How resources are distributed may be a potential issue. For 
example, will those who have the greatest need will be the only 
ones who could access the support services or anyone who is 
interested? If resources are limited, who gets to be prioritized? 


The Issues of Ethics and Privacy in Learning Analytics 


Slade and Prinsloo (2013) proposed a framework that can be used as a guide for higher 


education institutions to address ethical issues in learning analytics. This is a six-principle 


framework. 


1) Learning Analytics as Moral Practice. This must result in understanding rather than 
measuring (Reeves in Slade & Prinsloo, 2013). 

2) Students as Agents. Students should be collaborators and not only recipients of 
interventions and services. 

3) Student Identity and Performance are Temporal Constructs. Learning analytics provides 
only a snapshot view of a learner at a time and context. Data must eventually expire. 

4) Student success is a complex and multidimensional phenomenon. Our data is incomplete 
and can be vulnerable to misinterpretation and bias due to the nature of student success. 

5) Transparency on the purpose of using the data and how these will be protected. 

6) Higher education cannot afford to ignore data. This is particularly true if it helps an 
institution achieve its goals. 


Pardo and Siemens (2014) identified four principles that can be used to categorize the various 
issues that stem from privacy in learning analytics. These are transparency, student control, 
security, and accountability and assessment. Transparency can be applied in all the stages of 
learning analytics. The information must be clear to all the stakeholders how the analytics 
process is carried out. The type of information must be known to them. Student control, which 
covers the rights of the users to access and correct the data obtained about them. The right of 
access refers to being able to clearly identify who has access to the collected data, without this 
user trust would be affected. Finally, accountability and assessment suggest that each aspect 
must have an entity responsible for the proper functioning of its related components. Assessment 
refers to the ability of the institution to evaluate, review, and refine the entire process. 


Drachsler and Greller (2016) proposed a checklist named DELICATE which can be used by 
researchers, policymakers and institutional managers when implementing learning analytics 
solutions ‘‘to overcome the fears connected with data aggregation and processing policies” (p. 
96). This is an eight-point checklist derived from a thorough literature review, workshop with 
experts and several legal documents. Their checklist consists of the following: 


Determination, which answers the question of why a stakeholder wants to use learning 
analytics? It aims to determine what value does it add to the organization and to the data 
subjects. It also determines the rights of the data subjects. 

Explain, which suggests that institutions be open about their intention and objectives for 
accessing data. This includes being able to identify what data are needed and what 
purpose they serve. Furthermore, intentions about how long these data will be used and 
who gets to have access must be made clear. 

Legitimacy, the question of why accessors can have the data? It involves identifying the 
data sources that are already available and identifying why they are not enough, if 
applicable. Also, questions such as the ability to collect additional data must be 
answered. 

Involve, which highlights the importance of involving all the stakeholders and the data 
subjects in the data analysis process. This could include answering any privacy concerns 
of the data subjects as well as the training of any staff that would handle or have access to 
the data. 

Consent, which makes a binding agreement with the data subjects about the nature, use, 
and methodology of the data collection and analysis. This should be done prior to the 
data collection and must have clear and understandable consent questions (yes or no). 
Also, data subjects must have the option to opt-out of the data collection without 
consequences. 

Anonymize which ensures that individual information must not be retrievable. As much as 
possible, data should be aggregated to generate abstract metadata models. 

Technical which ensures the various procedures to guarantee privacy. This could be by 
continuously monitoring who can gain access to the data. If there are any changes to the 
analytics, the privacy regulation must be updated, and new consent must be asked. The 
data storage must comply with international security standards. 

External which indicates data collection and use may have different concerns to address 
if data will be collected or analyzed using assistance from external providers. Imperative 
that any external provider must comply with national and organizational rules. A contract 
which clearly states responsibility for data security must be made with any provider used 
in the project. Data collected must be used only for its original stated intent and not for 
some other purpose. 


Data mining 


What is Educational Data Mining? 

Educational Data Mining (EDM) is an emerging interdisciplinary research field which is 
concerned with developing, researching, and applying computerized techniques that will help 
make sense of a vast amount of educational data (i.e., captured from educational settings) with 
the hopes of detecting meaningful patterns (Romero & Ventura, 2013). The field of EDM sits at 
the intersection of fields of Computer Science, Education, and Statistics. The goal is to have a 
better understanding of how students learn as well as to determine in which setting students 
learn. This greater understanding will enable educators to gain insight and explain educational 
phenomena to improve educational outcomes. Another goal of EDM is to improve learning. 


However, such measures are not easily obtainable. Therefore, these are estimated through 
improved performances. Zaiane (2001) identified the goal of EDM as turning learners into 
effective and better learners. 


Romero and Ventura (2013) discussed the types of data being analyzed using EDM techniques 
and how they are not only limited to interaction data of individual learners (e.g., navigation 
behavior). These data could be collected from collaborating students (e.g., chat or discussion), 
administrative data (e.g., school or teacher), demographic data (e.g., gender or age), student 
affectivity (e.g., emotional states), among others. Furthermore, the typical characteristics of 
these data include multiple levels (e.g., assessment, question, or subject level), context, fine- 
grained (i.e., varying time resolution in terms of capturing data), and longitudinal data (e.g., 
data spans to multiple semesters or even years). 


Differences between Learning Analytics (LA) and Educational Data Mining (EDM) 

Siemens and Baker (2012) described differences in LA and EDM. They compared the two fields 
in terms of five different aspects, namely the type of discovery being prioritized, reductionist 
and holistic frameworks, origins, adaptation and personalization, and popular techniques and 
methods used. In terms of discovery, both LA and EDM aim to automate the discovery process 
with the use of visualizations and other methods. However, LA gives a greater focus on 
leveraging human judgement, while EDM focuses more on automated discovery. LA uses 
automated discovery to inform humans who make decisions, while EDM uses human judgment 
(e.g., experts) in the form of providing labels for classification. LA attempts to look at systems 
holistically by understanding them in their full complexity. On the other hand, EDM puts 
emphasis on reducing systems into their components, analyzing each of these components and 
understanding the relationships among them. LA has strong origins from the fields of semantic 
web, intelligent curriculum, outcome prediction, and systemic interventions. On the other hand, 
EDM has strong origins from the fields of educational software and student modeling, 
particularly in predicting course outcomes. In terms of adaptation and personalization, LA 
models are mostly designed to empower stakeholders (i.e., instructors and students) by 
informing them. On the other hand, EDM models are mostly designed for the use of automating 
adaptation in systems which do not have humans in the loop (e.g., intelligent tutoring systems). 


Regarding the techniques and methods, LA usually uses social network analysis, sentiment 
analysis, influence analytics, discourse analysis, learner success prediction, concept analysis, 
and sensemaking models. Meanwhile, EDM typically uses classification, clustering, Bayesian 
modeling, relationship mining, discovery with models, and visualization. Ferguson (2012), in her 
discussion of the state-of-the-art of learning analytics, identified the central theme of the 
research fields as both fields started to mature. LA is focused on the educational challenge: How 
can we optimize opportunities for online learning? EDM is focused on the technical challenge: 
How can we extract value from these big sets of learning-related data? 


Types of Educational Environments of EDM Research 


EDM researchers use data that come from either the traditional education or the computer-based 
education environments. In the educational system, traditional education is the most widely 
used. This environment involves face-to-face contact between the teacher and the students, 
which typically consists of lectures, individual work, and class or small group discussions. Such 
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environments include infant/preschool education, primary/elementary education, secondary 
education, higher/tertiary education, and alternative/special education. Meanwhile computer- 
based educational environments use computers to provide direction or to instruct students, such 
as learning management systems (LMS), intelligent tutoring systems (ITS), adaptive and 
intelligent hypermedia systems (AIHS), test and quiz systems, and other types (e.g., wikis, 
forums, virtual reality). Romero and Ventura (2013) noted that traditional education may use 
computer-based educational systems to complement their face-to-face sessions. 


Topics of Interest in EDM 

Romero and Ventura (2013) enumerated the various topics of interest in the educational data 
mining research community. These topics include: developing generic frameworks and 
methods, mining educational data (e.g., assessment or interaction data), educational process 
mining (i.e., extracting process-related knowledge from event logs), data-driven adaptation and 
personalization, improving educational software, evaluating teaching interventions, detecting 
emotion, affect, and choice, integrating data mining and pedagogical theories (i.e., use existing 
educational and psychological knowledge to better focus research), improving teacher support, 
replication studies (i.e., application in a new domain), and best practices. 


Popular Methods of EDM 

Romero and Ventura (2013) surveyed the state-of-the-art of EDM research and summarized the 
various methods that were used by researchers. The table below enumerates these methods and 
briefly describes them. Romero and Ventura (2013) noted that the distillation of data for human 
judgment, discovery with models, knowledge tracing, and nonnegative matrix factorization are 


mostly prominent in EDM research. 


Prediction 


Clustering 


Outlier Detection 


Relationship Mining 


Social Network Analysis 


Process Mining 


Text Mining 


Distillation of Data for 


Human Judgement 


Discovery with Models 


Knowledge Tracing 


Nonnegative Matrix 
Factorization 


Infer a target attribute based on a set of attributes (or combination of). 
This includes classification, regression, or density estimation. This 
approach is popular to forecast student’s performance. 


Identify groups of students that are like one another. This could be due 
to their similar learning and interaction patterns. 


Identify data points that are significantly different from others, usually, 
these values are either too small or too large as compared to others. 
This can be used to detect students with learning difficulties. 


Identify the relationship between variables and encode them in a form 
of rules. Popular approaches include association rule mining, 
sequential pattern mining, correlation mining, and causal data mining. 
This could be used to identify relationships between student’s behavior 
patterns and their learning difficulties. 


Measure relationships among entities in a networked context (i.e., 
nodes and links). This can be used to analyze the structure and 
relations in tasks that allow for collaboration and interactions. 


Using event logs captured by information systems, process-related 
knowledge is extracted with the goal of coming up with a clear visual 
representation of the whole process. 


Extract useful information from text data. This usually involves text 
categorization, text clustering, concept/entity extraction, production of 
granular taxonomies, sentiment analysis, summarization of document. 
This could be used to analyze the content of discussion forums or chats. 


The goal is to represent data in a way that is easily comprehensible. 
This could be achieved through summarization, visualization, and 
interactive interfaces to highlight information that can be used for 
decision making. EDM does a good job at this, as large amounts of data 
can be presented at once. This would enable instructors to visualize 
and analyze the course activity of students along with their usage. 


Uses models previously validated as a component of another analysis. 
Usually, these are used in predictions or clustering. This enables the 
analysis of existing research questions across a variety of contexts. It 
could also identify relationships between the behaviors of students and 
their characteristics. 


Estimate the mastery of a student on a skill, which has been used in 
effective cognitive tutors. This uses cognitive models that map a 
problem-solving item to a particular skill being assessed. Logs of 
correct and incorrect answers are then used as basis or evidence of 
their knowledge for a skill. 


A technique that enables interpretation in terms of Q-Matrix (or transfer 
model) in a straightforward manner. 


Common Applications of EDM 

The most common application of the EDM approach is predicting students’ performance, which 
is also the oldest and the most popular approach. However, more studies involving the 
application of EDM techniques to solve other problems have been starting to emerge. Romero 
and Ventura (2013) listed several, such as user and student modeling through the development 
and tuning of cognitive models which represent their skills and declarative knowledge (Frias- 
Martinez, Chen, & Liu, 2006). Findings show the construction of courseware which would benefit 
instructors and developers of learning content (Garcia, Romero, Ventura, Gea, & De Castro, 
2009). Another example is parameter estimation where parameters to probabilistic models are 
inferred from the data with the aim of predicting the probability of an event of interest to happen 
(Wauters, Desmet, & Van Den Noortgate 2011). Finally, Romero and Ventura (2013) provided a 
list of examples of educational data mining tools that have been used by researchers in the field. 
Some examples include SNAPP, DataShop, EPRules, among others. 


Stealth Approaches to Assess Students in Online Learning Environments 


Reeves (2000) differentiated assessment from evaluation. Assessments are for people, and 
evaluations are for things. However, it is easy to confuse and interchange both. Reeves (2000) 
discussed three major directions for integrating alternative approaches for assessing in online 
learning environments in higher education: cognitive assessment, performance assessment, and 
portfolio assessment. The cognitive assessment focuses on measuring the higher order thinking 
abilities of students, achieved through means such as concept mapping. Performance 
assessment can be done by looking into the learner’s ability to apply knowledge in realistic 
contexts, done by requiring students to demonstrate their capabilities directly through product 
creation or through engagement. Finally, a portfolio is where the work of the student is stored 
over time so that it can be reviewed with respect to both process and product. 


Stealth Assessment. 

Shute (2011) argued that assessments can be integrated into the learning process which could 
enable us to extract evidence and react in meaningful ways. In this approach, automated scoring 
and machine-based reasoning can be leveraged. The idea of integration led to the development 
of stealth assessment. Shute and Kim (2014) formally defined it as “an evidence-based approach 
to assessment where the tasks that students are engaged with are highly interactive and 
immersive” (p. 135). These environments could be in the form of video games or other 
computer-based instructional systems. Throughout this process, a person’s progress is 
continually being tracked and data related to the progress are being collected, and immediate 
feedback is provided. For example, in the context of computer games where people explore 
simulated worlds, software can be used to track and collect data regarding the user’s progress. 
The goal is to predict appropriate challenges so that these challenges can be provided to 
learners based on what the system knows about the user so far (Sharples, 2019). Stealth 
assessment is a special approach to formative assessment (Shute & Kim, 2014). Stealth 
assessment and instructional design share a common goal which is to coherently align learning 
objectives with how they are measured. 


Principles of stealth assessment. Sharples (2019) enumerated key principles of stealth 
assessments. The software that analyzes the activities of students within a computer game or 


simulation should be able to continually adjust to match the challenges to the performance of the 
student. The testing should be part of the game and should not be separated. In effect, the 
learner's anxiety is reduced because of the blurring distinction between assessment and 
learning. Finally, the system should build models dynamically which estimates the student’s 
ability and competency. 


Stealth assessment design. 

Competency learning is the underlying pedagogy of stealth assessment (Sharples, 2019). Shute 
(2011) discussed how to design effective stealth assessments. Evidence-centered Design (ECD) 
is believed to be a successful method to develop stealth assessment games. In this method, 
game designers identify what knowledge, skills, and competencies to assess. Usually, these 
competencies are not easily directly assessed. To address this, the behaviors, and interactions of 
the students with the system are often used as evidence. The game designer then builds different 
measures of success and failure into the game which will then be linked together to form a 
network of probabilities of the learner having gained the desired skill or reached the required 
competency. These inferences on competency states are stored in a dynamic model of the 
learner. 


Issues surrounding stealth assessment. 

One ethical issue regarding stealth assessment is how it claims to provide students an 
entertaining game but, their progress is being monitored and their problem-solving skills or 
creativity are being assessed (Sharples, 2019). Furthermore, as research in stealth assessment is 
still at an early stage, the possibility of adopting a set of general methods of design is still 
unclear. For instance, is it possible for methods implemented for a game or topic to be reused in 
others (Sharples, 2019)? 


Blended Learning, Data Analytics, and Educational Data Mining 


Learning analytics have been used to predict student success in courses and LA can be used to 


inform instructors and adjust learning designs for the betterment of course content and delivery 
(Foung & Chen, 2019). Foung and Chen (2019) found that some students engaged in the online 
component of the blended course early in the term but only did the minimum required amount of 
online work, while other students accomplished far beyond the minimum online activities and 
accessed the online information after the course was completed. In their research, Foung and 
Chen (2019) found that the total number of attempts to complete an activity and the student’s 
performance on individual online activities was predictive of the student’s final course grade. 
Similarly, Lu et al. (2018) used LA to predict students’ final course grade after only one-third of 
the course had been completed. 


That said, traditional LA may not always be appropriate for enacting change in blended learning 
endeavors. Sansone and Cesareni (2019) suggested that LA be fashioned around learning 
theories to measure the critical aspects of active online interactions. The Social Learning 
Analytics model uses such factors as the number of clicks, discussion forum participation, and 
formative assessment on computer-assisted technology to monitor and improve learning 
outcomes (Sansone & Cesareni, 2019). The Social Learning Analytics model is not used 
ubiquitously, and Sansone and Cesareni (2019) suggested that developers, researchers, and 
instructors work together to standardize the approach to learning analytics in blended 


environments with the goal of the data being more capable of accurately influencing training 
interventions, personalized automatic feedback, encourage student reflection, and highlight 
best practices in these hybrid environments. 


Learning analytics are of no concrete value if they cannot generate actionable information (van 
Leeuwen, 2019). Van Leeuwen (2019) found that teachers do not always know what actions to 
take from the LA data, but that instructors did find the reports caused them to begin 
conversations with students and to diagnose and intervene during learning activities. Van 
Leeuwen (2019) suggested that LA reports could better support instructors if the report 
contained not only raw data but suggestions on implementing interventions. 


One suggestion that can be implemented by instructors to better support students is the use of 
appropriate and timely personalized feedback (Pardo, Jovanovic, Dawson, Gasevic, & Mirriahi, 
2019). Personalizing feedback can be especially challenging in large student cohorts (Pardo et 
al., 2019). Pardo et al. (2019) developed an algorithm that helped instructors tailor email 
messages to students each week throughout a course based on sets of pre-written replies to 
students about how they could adjust their study to improve learning. The algorithm was based 
on the number of correct responses on summative exercises within a range (Pardo et al., 2019). 
Results demonstrated a positive association between the messages and the learners’ satisfaction 
with the feedback and students’ academic performance on the midterm examination. 


Park, Yu, and Jo (2016) found that it was possible to track the extent of an institution’s 
incorporation of a blended learning plan through educational data mining. The Korean institution 
studied had expressed policies for the implementation of blended learning; however, Park et al. 
(2016) were able to uncover that more structural supports were needed to fully change the 
culture of the institution and incorporate blended learning to the extent that was proposed. Park 
et al. (2016) demonstrated that online usage was extremely active for a small number of courses, 
but most of the institutions online offerings presented with a low LMS usage pattern. The EDM 
was able to uncover the discrepancy between policies and practice and the institution was able 
to disclose the results to the stakeholders so that plans for implementation of the blended 
learning goal could begin (Park et al., 2016). 


Visualization 


Motivations for Visualizing Data 

Ruiperez-Valiente, Mufioz-Merino, Leony, and Kloos (2015) suggested that there are two main 
approaches to learning analytics that can be employed to make sense of the vast amount of 
learning data. Systems can be built which automatically processes the data, an example would 
be intelligent tutoring systems (ITS) or recommender systems. Another approach is through 
direct reporting to stakeholders (i.e., visualization). Information visualization uses interactive 
visual representations to amplify cognition (Card, Mackinlay, & Shneiderman, 1999). The ability 
of humans to recognize or discover patterns from visualizations (e.g., trends, outliers, clusters, 
gap) forms the basis of information visualization. By adding an interactivity component to 
learning, instructors facilitate exploratory data analysis. 


Benetits of visual data exploration 

Learner participation must be considered for data mining to be effective. This is where human 
knowledge and creativity comes into play with the computational power of modern computers. 
This process is called visual data exploration, where individuals are presented with visual forms 
of the data. Through this, they are asked to provide insight and draw conclusions from it. 
Furthermore, it is also possible for them to interact with the data, a process known as hypothesis 
generation. 


Shneiderman (1996) provided a visual exploration paradigm popularly known as the Information 
Seeking Mantra (ISM), which provides a guideline on how to design effective visualizations, 
(e.g., “Overview first, zoom and filter, then details-on-demand”, p. 337). ISM stresses the 
importance of providing an initial overview to users to give them opportunities to detect 
interesting patterns. To detect and analyze patterns, users must be able to drill down and access 
details of the data. Keim (2002) suggested keeping the overview available to the user while 
focusing on the subset using another visualization technique. One such approach is the use of 
distortion, where the overview is distorted to focus the user on the subset. 


Shneiderman (1996) enumerated seven tasks that can be done in visual exploration: 


e overview (a complete picture of the collection which contains a movable field-of-view 
box) 
zoom (focus on items of interest, part of the collection, ideally preserve the sense of 
position and context, can be one dimension at a time or altogether) 
filter (remove any items that are not of interest, in essence do this dynamically, ideally 
less than 100 ms) 
details-on-demand (the ability to be able to provide details of items or group of items 
when selected) 
relate (ability to view relationships among items) 
history (ability to keep track of actions to facilitate undo or redo, or even further 
refinement) 

e extract (retrieve parameters of query or subset of the collection and export them) 


What types of data can be visualized? 

Shneiderman (1966) enumerated seven data types: 1-dimensional, 2-dimensional, 3- 
dimensional, temporal, multi-dimensional, tree, and network. Specific to visual data mining, 
Keim (2002) identified six data types to be visualized: 1-dimensional data, two-dimensional data, 
multi-dimensional data, text and hypertext, hierarchies and graphs, and algorithms and 
software. 


What are the different techniques that can be employed fo visualize data? 
Various techniques to display data include: 

Standard 2D/3D 

Geometrically transformed displays 

Iconic displays 

Dense pixel displays 

Stacked displays 
Interaction and distortion techniques include: 


Dynamic projects 

Interactive filtering 

Interactive zooming 

Interactive distortion 
Interactive linking and brushing 


The impacts of visualization on students. 

Grissom, McNally, and Naps (2003) conducted a multi-university study that investigated the 
impact of algorithm visualization on student learning. The researchers identified the effects of 
different levels of algorithm visualization: not seeing any visualization, simply viewing 
visualizations for a short period in the classroom, and interacting directly with the visualizations 
for an extended period outside the classroom. Findings suggested that as the student 
engagement level increased, learning increased. Learning in this study was measured by 
computing for the learning gain (post-test minus pretest scores) in the context of an introductory 
computer science course. 


Falakmasir, Hsiao, MazzolaGrant, and Brusilovsky (2012) investigated the impact of visualization 
on students’ performance in a C Programming Course. The authors used a system called 
KnowVis, finding that students from a group who had visualization were more engaged in 
learning activities. Furthermore, these visualization students performed better in self-assessment 
quizzes, which may have been due to them being conscious or aware of their performance. The 
visualizations also fostered competition among their peers, resulting in students having better 
accuracy. 


Dashboards 


Jivet, Scheffel, Specht, and Drachsler (2018) suggested that learning analytics could bridge the 
gap between learning sciences and data analytics. Educators and researchers could derive 
meaning from the vast amount of data captured by online learning environments. One popular 
learning analytics intervention is the learning dashboard. Few (2013) defines a dashboard as ‘“‘a 
visual display of the most important information needed to achieve one or more objects that have 
been consolidated on a single computer screen so it can be monitored at a glance”’ (p. 26). Few 
(2013) identified some of the essential characteristics of these dashboards, such as they are 
visual displays that display information that is needed to achieve specific objectives. Virtual 
displays usually fit on a single computer screen and are used to monitor information briefly. Yoo, 
Lee, Jo, and Park (2015) defined an educational dashboard (or learning dashboards, learning 
analytics dashboard) as an umbrella term which is “a visualized and intuitive display derived 
from the results of educational data-mining for the purpose of supporting students’ learning and 
performance improvement” (p. 147). 


Dashboards support learning or teaching by visualizing learning traces for learners and teachers 
(Verbert, Duval, Klerkx, Govaerts, & Santos, 2013) and provide a current and historical state of a 
learner, which allows for flexible decision making (Few, 2006). Furthermore, Schwendimann et 
al. (2016) define dashboards as “a single display that aggregates different indicators about 
learner(s), learning process(es) and/or learning context(s) into one or multiple visualizations” 
(p. 37). The aim of these dashboards is mainly to provide feedback on learning activities and to 
support reflection and decision making (Klerkx, Petter Straete, Kvam, Ystad, & Butli Harstad, 
2017). Also, dashboards help to keep students engaged and motivated, which in turn could lead 
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to a lower dropout rate. Dashboards are considered to be a specific class of “personal 
informatics” applications (Li, Dey, & Forlizzi, 2010), which allows users to collect various aspects 
about themselves to help in understanding their status (Li, Dey, Forlizzi, H6ok, & Medynskiy, 
2011). Having this understanding leads to an improvement of self-knowledge through the review 
and analysis of their personal history. This field is still new, therefore research on principles are 
currently limited (Yoo et al., 2015). 


Types of Learning Dashboards 

Learning dashboards can be categorized into three types, namely those that support traditional 
face-to-face lectures (e.g., those that can be used to inform instructors for them to adapt their 
strategies based on the needs of their students), those that support face-to-face group work and 
classroom orchestration (e.g., those that visualize activities of both individual and group of 
learners), and those that support online or blended learning (e.g., those that support awareness, 
reflection, sense-making, and behavior change) (Verbert et al., 2014; Klerkx et al., 2017). 


What information can be incorporated in learning dashboards? 
Verbert et al. (2014) identified some of the potential information that can be incorporated into 
learning dashboards. These include: 


e 6 Artifacts produced by learners (e.g., blog posts; those that end up in project portfolio) 
e Social interaction (face to face, group, blog comments) 
e Resource use (views of videos) 
Time spent (for teachers to identify students at risk; students to compare effort amongst 
peers) 
Test and self-assessment results (to indicate learning progress) 


What are the steps to get started when designing information visualization systems? 
Klerkx et al. (2017) outlined the steps when developing information visualization systems: 


1) To understand the visualization goals, determining why visualization is necessary will 
shape the approach of the visualization. Also necessary is identifying for whom the 
visualization is intended (e.g., teacher, student, administrator) and how best achieve the 
projects’ goals. 

Acquire and pre-process data. According to visualization experts, this step takes 80% of 
the time and effort in setting up a visualization system. In this step, raw data is acquired, 
analyzed, and cleaned, as necessary. Furthermore, this step may also involve filtering 
out data that are not relevant to the main question being addressed. 

3) Mapping a design which focuses on identifying the best visualization to represent the 
data that would be fitting for the target audience. Also, the goal should be taken into 
consideration in this step. 

4) Documenting where explicit protocol is written to provide a rationale of the project 
decisions made and what alternatives could have been chosen. Also, a discussion on 
how the visualization evolved during the initial phase until the current state should be 
included. As the process of visual analysis is an iteration of view creation, exploration, 
and refinement, the next step is to add interaction techniques. This could include 


brushing and linking, histogram sliders, zoomable maps, dynamic query filter widgets, 
among others. 
5) The last step is to evaluate continuously. 


Alternatively, Verbert et al. (2014) suggested three techniques that could be used to evaluate 
such information visualization systems. These are effectiveness (e.g., such as an improvement in 
engagement, retention rates, self-assessment, course satisfaction, or post-test results), efficiency 
(e.g., time spent by the teacher or the learner), and usability and usefulness (e.g., are the 
teachers able to identify those students who are at risk; are the students able to assess their 
performance in the course). 


What are some issues of learning analytics dashboards? 

In a systematic review done by Matcha, Gasevic, and Pardo (2019), it was found that existing 
learning analytics dashboards (LADs) are rarely grounded in learning theory, cannot be 
suggested to support metacognition, do not offer any information about effective learning tactics 
and strategies, and have significant limitations in how their evaluation is conducted and 
reported. 


In addition, Verbert et al. (2013) noted that there is a need for more research on investigating the 
real impact of dashboards for improving learning or teaching. The lack of empirical evaluations 
is mainly due to dashboards being part of an exploratory investigation on a system as most of 
these systems were proof-of-concept (Schwendimann et al., 2016). Furthermore, longitudinal 
studies with these dashboards should also be explored to know the extent of how dashboards 
can affect the behavior of students or teachers. 


How do we evaluate LADs? 

Verbert et al. (2013) examined the characteristics of 15 different dashboard systems in terms of 
target users (teachers or students), tracked data (time spent, social interaction, document and 
tool use, artifacts produced, and exercise results or quizzes), and evaluation methodology 
(usability, usefulness, effectiveness, and efficiency). Among these 15 systems, only 10 were 
evaluated with teachers or students, or both. In terms of effectiveness and potential impact, only 
four systems have been evaluated. One of which is Course Signals (Arnold & Pistilli, 2012) which 
was evaluated across multiple academic years and on a large scale. As previously stated, Arnold 
et al. (2012) found that the retention rate of those who used the system at least in one course is 
significantly higher than those who did not use the system at all. This was the only system that 
was able to demonstrate an actual impact of dashboards on learning. The other systems were 
evaluated on laboratory-controlled settings with fewer participants, essentially highlighting the 
usefulness of dashboards. 


In their study, Verbert et al. (2013) also proposed a process model based on personal 
informatics applications. The four stages were: awareness (if people are aware of the visualized 
data), reflection (do people assess their performance by reflecting on the data), sense-making 
(when people answer the questions from the reflection level and create new insights), and 
impact (when people change their behavior). Yoo et al. (2015) were able to compare this 
process model with Kirkpatrick’s four-level model. Yoo et al. (2015) reviewed 10 major 
educational dashboards that have been introduced in academic journals and international 


conferences. In this study, they were able to develop an evaluation framework that was based on 
Kirkpatrick’s four-level model (which is usually used in training program and e-learning 
courseware) and Few’s principles of dashboard design. Yoo et al. (2015) were able to come up 
with detailed indexes based on the MECE (mutually exclusive and collectively exhaustive) 
principle. Using this framework, they evaluated the dashboards they identified. Yoo et al. (2015) 
found that, 1) students and teachers were more informed of their activities in the learning 
environment because of the dashboard, 2) in the cases they surveyed, social network (behavior 
in discussion forum or content or message exchange), at-risk student prediction (alerting those 
who might fail), and message analysis (summarized as tag cloud) were attempted, 3) only a few 
case studies considered dashboard design principles when they were designed, and 4) only a 
few cases conducted evaluation at the four levels, some did not, therefore the effectiveness were 
not investigated and proved. They also note that it is important to identify which student 
information is valuable to show (e.g., login trend, scores). 


Jivet et al. (2018) identified six levels or criteria used for evaluation in the 26 papers they 
reviewed. These criteria were metacognitive, cognitive, behavioral, emotional, self-regulation, 
and tool usability. They also identified papers that targeted the first five competencies and those 
that evaluated changes in these competencies, known as coverage, behavioral, cognitive, and 
emotional (only four) are evaluated in most cases where dashboards are designed to support 
such competencies. There is a large percentage that missed evaluation on metacognition and 
self-regulation. Jivet et al. (2018) found that most dashboards under metacognition aim to 
support awareness and reflection; however, only half assessed whether there was indeed an 
impact. 


What are some recommendations when designing and evaluating dashboards? 

Jivet et al. (2018) concluded with a list of recommendations for designing and evaluating 
dashboards. These were the compiled insights they gathered in their literature review. For 
instance, when designing dashboards, pedagogical tools that could catalyze changes in the 
cognitive, behavioral, and emotional competencies through the enhancement of awareness and 
reflection should be considered. Furthermore, the design decisions must be grounded in 
learning sciences principles. It is also important to keep in mind that the effect of the dashboard 
may not be the same for all users. The group that benefits the most must be identified, along with 
how to customize the dashboard so that the same support can be provided to all users. 


Comparing users with their peers in the dashboard should be used with caution. The dashboard 
should be integrated seamlessly into the online learning environment. In terms of evaluation, it is 
important to first assess dashboards by looking at the goals of the dashboard, followed by its 
impact on learners’ affect and motivation, and lastly by its usability. In evaluating in terms of 
usability, it should not be limited to looking at whether it is usable or useful. Rather, usability 
should be assessed as the ability of the users to trust the system (i.e., transparency) or whether 
the learners agree with it and how it is interpreted. When evaluating dashboards data 
triangulation (self-reported data, tracked data, assessment data) must be used to validate its 
effects, Design features that rely on educational concepts should be assessed. In terms of 
assessing the impacts of the dashboard on the learner, validated measurement instruments must 
be used. 


Blended Learning, Visualization, and Dashboards 
Our literature search did not locate any studies that specifically delineate the effectiveness of 
dashboards and visualization techniques specific to blended learning contexts. 


Cheating in Online Systems 
Dishonesty in Online Systems 


In his work, Rowe (2004) identified the typical types of problems or issues that can occur when 
assessing learners online. He noted how the issue of plagiarism has often been explored but not 
in the context of dishonesty in online assessments. Proposed measures on how to counteract 
academic dishonesty include preventing students from obtaining advance answers to 
assessments. This could happen if one learner takes a screenshot of the questions for other 
students or if students view the screenshot that was taken. Assessing students all at one time is 
challenging. Another issue is the possibility of unfair retaking or grade-changing for 
assessments. This could be in the form of a made-up excuse where the student lost electricity 
while taking the exam. Lastly, the issue of receiving unauthorized help while taking the 
assessment is possible. Students can arrange for “consultants” to help them with difficult 
questions. Confirming the identity of students--whether they really are who they say they are 
when taking an assessment--is not easy. Moten Jr, Fitterer, Brazier, Leonard, and Brown, (2013) 
identified some other ways for students to cheat in online environments, including waiting for 
answers, as some instructors offer flexibility in terms of when to take exams. There are certain 
cases where multiple monitors are used (one is for searching and the other is for assessment 
taking). Another is the claiming of fraudulent error messages, where students claim to have 
encountered an error while using the system. This may or may not be true. However, the main 
objective is for the student to have more time to prepare for the exam. Finally, there is collusion 
where students would work together such as in essay plagiarism or purchasing answers. 


Countermeasures to curb online cheating. 

Research to explore how to counter these dishonest behaviors has been conducted. One 
common countermeasure is the use of statistical analysis to detect common errors or patterns 
(D'Souza & Siegfeldt, 2017; Moten et al., 2013; Rowe, 2004). This could be performed easily on 
multiple-choice or true/false questions where the distribution is analyzed and the similarity 
between students is evaluated. However, this approach must be used with caution as it should 
not be used to establish guilt as some students may be innocent victims (Rowe, 2004). Other 
countermeasures include making the assessment a learning experience while ensuring that it is 
not either too easy or too hard. Extremely hard assessments tend to encourage cheating. 
Another countermeasure is to use constructed-response test formats (e.g., programmed 
calculation to obtain answers) and to use varied test formats. 


Automatically detecting cheating behaviors in online systems 
Chuang, Craig, and Femiani (2015) investigated the ability to use personal or situational factors 
of students to predict the intentions of cheating. In this controlled laboratory experiment, they 


were able to identify two factors that could potentially indicate intentions of cheating in online 
systems, namely the time delay (positive predictor) and the student’s certainty on a question 
(negative predictor). They highlight how these factors could be easily and objectively obtained 
in real-time as opposed to the conventional approach of the use of questionnaires. The use of 
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questionnaires is prone to self-report bias, either by overreporting or underreporting. This study 
highlights the potential of incorporating automated analysis of video data (e.g., looking at facial 
expressions and associating them to affective states to quantify certainty). In a follow-up study, 
Chuang, Craig, and Femiani (2017) further investigated other factors, taking into account how 
unnatural it is for students to rate their certainty on questions in a real-world setting, and this 
certainty is a form of self-report. In this study, they were able to find that the amount of head 
movement variation with respect to the monitor, along with time delay were positive predictors 
of cheating behaviors during online exams. Of the two factors, time delay had a higher 
predictive power. This study demonstrates the possibility of building a proctoring system that 
could help automate the flagging of suspicious students. Chuang et al. (2017) cautioned about 
the validity of the findings in the real-world settings as both studies were conducted in 
laboratory settings, so the classification accuracy is still unknown. 


Another approach proposed by Young, Davies, Jenkins, and Pfleger, (2019) is the use of 
keystroke dynamics to create Keyprints to authenticate individuals in online courses. This proof- 
of-concept study attempted to address the challenge of verifying a student’s identity in an online 
learning environment (i.e., whether they are really who they say they are), most especially when 
taking assessments. This is achieved by leveraging the typing behavior of the student (Monrose 
& Rubin in Young et al., 2019). This is considered a cost-effective approach to improve online 
assessment security. Keystroke dynamics could include data such as dwell time (amount of time 
a key is depressed) and transition time (time in between the time when the previous key is 
returned to its original position and the next key is depressed). Keystroke dynamics could also 
include typing speed and the number of errors. In their study, Young et al. (2019) found that 
keyprints are unique but a full keyprint signature may be more accurate than a reduced one. 
They noted a limitation of their system where it could not detect who typed it, however, 
keyprints could indicate with a high degree of probability if the typer was not the intended user 
on at least 70% of the data points. However, further research is needed to explore whether 


factors such as if using a different keyboard affects the keyprint matching. 


Amigud (2018) noted that to be able to maintain academic integrity in online assessments, two 
distinct layers are needed to be confirmed, physical and behavioral. These two comprise the 
identity of the learner. They found that most of the papers focus on the detection or deterrence of 
cheating behaviors. Some of the strategies for identity assurance that they found include: identity 
document verification (e.g., use of photographic identifications), password-based authentication 
(i.e., use of password in combination of other techniques), challenge question-based identity 
verification (i.e., questions that only the rightful owner knows), biometric-based identity 
verification (i.e., uses physiological and behavioral traits particular to the individual), and multi- 
factor authentication (i.e., a combination of multiple factors such as discussed previously). Some 
strategies for authorship assurance can aid in detecting cheating. These include plagiarism 
detection tools (i.e., use of tools to detect duplicate contents), proctoring (i.e., use of remote 
proctoring for supervision), behavioral biometrics (i.e., uses behaviors to determine 
consistency), instructor validation (i.e., instructor knowing their students and their styles), 
computer lockdown and network monitoring (i.e., restriction of external resources and 
monitoring for possible collusion), instructional design (i.e., designing the learning materials 
and activities to decrease the benefits of cheating), and policy (i.e., honor codes and academic 
integrity policies). There are also other methods discussed which are aided by data analytics 


(e.g., statistical tools or the use of machine learning techniques). Identifying whether such 
security approaches are effective is still to be explored. 


Applications of Technology for Supporting Distributed Learning 
Intelligent Systems and Personalized learning 


The term “personalized learning” has been around for years, but the adoption of personalized 
learning has increased significantly due to the rapid development of technology and the massive 
data from the ubiquitous Web (Pane, Steiner, Baird, & Hamilton, 2015; Jenkins & Keefe, 2002). 
There is no universal definition of personalized learning, but personalized learning was broadly 
defined as, a student-centered approach that primarily focuses on supporting a student’s needs 
and interests to optimize their learning experiences and motivate them in the learning process 
by harnessing the power of technology (Song, Wong, & Looi, 2012; Bulger, 2016; Basham, Hall, 
Carter Jr, & Stahl, 2016; Thyagharajan & Nayak, 2007). Personalized learning is not equal to 
customization, in fact, customization is only one type of personalized learning which refers toa 
customized interface (Bulger, 2016). The other types of personalized learning include interactive 
learning environments, flexible scheduling and pacing, and authentic assessment (Jenkins & 
Keefe, 2002). 


Personalized learning can take place in the traditional face-to-face learning settings, as well as in 
the technology-enhanced learning settings. In the traditional settings, personalized learning 
usually means personalized instruction were teachers tailor the curriculum programs to allow 
student-driven learning (Nandigam, Tirumala, & Baghaei, 2014). With the development of 
technology, personalized learning has evolved into a more powerful approach which allows 
students to take full control of their learning and become informed data-driven learners 
(Nandigam, et al., 2014). For example, the use of an intelligent learning system makes 
personalized learning possible and accessible to learners. Most of the current research on 
personalized learning relies heavily on technology-enhanced learning, and this direction will be 
the focus of this review. 


In the context of technology-enhanced learning, learning engineers exploited the modeling 
affordance of computer and artificial intelligence (AI) techniques, and developed online learning 
systems that provide individual support during their learning process, as a human tutor would 
do in a traditional face-to-face setting (Magnisalis, Demetriadis, & Karakostas, 2011; Mitrovic, 
Martin, & Suraweera, 2007). Such tutoring systems are known as Intelligent Tutoring Systems 
(ITS). 


According to Nwana (1990), the four basic components that the typical ITSs include are: 1) the 
expert knowledge module, 2) the student module, 3) the tutoring module, and 4) the user 
interface module. These four basic components composed a general architecture of a typical ITS. 
In particular, the expert module refers to the domain knowledge that encompasses the concepts, 
facts, rules, and strategies of the domain. This module serves as an expert resource for students, 
and it requires explicit and exhaustive representation of such knowledge in AI. Secondly, the 
student module refers to the dynamic representation of students’ learning progression and 
learning outcomes (Ahuja & Sille, 2013; Nwana, 1990). Specifically, as the system traces more 
student learning data (e.g., their cognitive and affective states, etc.), it will automatically predict 
and adapt itself to meet the students’ needs. Next, the tutoring module is an actionable output 
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from the integration between the domain and student models. The system pays attention to the 
student model and effectively utilizes the knowledge in the domain model to generate the 
appropriate pedagogic activities (Ahuja & Sille, 2013). Therefore, it also refers to the teaching 
strategy or the pedagogic module (Nwana, 1990). Lastly, the user interface model is the 
interacting front-end communicating component of the ITS between the student and the system. 
It translates the tutor model into an understandable interface language for student use (Ahuja & 
Sille, 2013; Nwana, 1990). 


In recent years, progress has been made towards providing adaptivity and personalization in 
technology-enhanced learning with advanced technology (e.g., machine learning and natural 
language processing) and the evolving of learning theory and cognitive research. ITS have 
gradually developed into providing adaptivity and personalization, also known as 
adaptive/personalized intelligent tutoring systems (Ahuja & Sille, 2013). 


Of the four components of ITSs, student model is considered the building block in the realm of 
ITSs because not only can it make tutoring module understand students’ learning behavior, but 
also help the tutor to make the appropriate actions based on the diagnosis of students learning 
behavior (Kurup, Joshi & Shekhokar, 2016). Kurup and colleagues (2016) reviewed five student 
modeling techniques and concluded that Bayesian Knowledge Tracing (BKT) is the most widely 
accepted student modeling technique because of its accuracy in inferring, predicting, and 
analyzing a student’s proficiency in each skill. BKT carefully analyzes a student’s mastery of each 
skill from a personal performance history (Gong, Beck & Heffernan, 2010). 


Kulik and Fletcher (2016) conducted a meta-analytic review with respect to the effectiveness of 
ITSs. They first made a distinction between ITSs and Computer-Assisted Instruction (CAI) by 
pointing out the “intelligent” features of ITSs compared to CAI. For example, CAI focuses on 
programming instructional feedback for guiding learners to find the right answers to the 
questions. Rather, ITSs emphasized that the utilization of artificial intelligence and cognitive 
theory to create hints and feedback as needed to assist students to solve problems in the domain 
(VanLehn, 2011). From Vanklehn’s (2006) point of view, the most prominent features of ITSs that 
separated it from CAI are that ITSs give learners both end-of-problem support (e.g., giving a 
learner feedback/suggestions on a problem solution and appropriate new problems to solve) 
and just-in-time support (e.g., giving prompts, hints and other feedback while a learner is 
working on a problem). However, CAI only provides learners with end-of-problem support. 


The effectiveness of ITSs has been shown through multiple meta-analyses (Steenbergen-Hu & 
Cooper, 2013, 2014; VanLehn, 2011; Ma, Adesope, Nesbit & Liu, 2014; Slavin, Lake, & Groff, 
2009). While all have found some positive benefit, the extent of that benefit seems to vary. For 
example, as for effectiveness, Steenbergen-Hu and Cooper (2014) found that ITSs raised test 
scores overall by 0.35 standard deviations, while VanLehn (2011) found that the average ITS 
effect improved the tests scores by 0.58 standard deviations, and Ma et al. (2014) found that the 
average ITS effect improved test scores by 0.43 standard deviations. In contrast, two recent 
reviews in relation to mathematics learning specifically reported that the use of ITSs had no 
significant improvement in school performance. Slavin et al., (2009) found that the Cognitive 
Tutor Algebra raised students’ test scores by the average of 0.12 standard deviations, and 
Steenbergen-Hu and Cooper (2013) found a difference of only 0.05 standard deviations. 


To address this lack of consensus in terms of an ITS’s effectiveness, Kulik and Fletcher (2016) 
reviewed 50 reports that described evaluations that meets their requirements for their meta- 
analysis. Their findings consisted of three influential factors: (a) the type of posttest (e.g., local 
developed tests that focus on problem solving versus standardized multiple-choice tests that did 
not emphasize problem solving), (b) the condition of the control groups (e.g., conventional 
control group versus nonconventional control group), and (c) the adequacy of ITS 
implementations (e.g., the different backgrounds and teaching styles from the teachers who 
implemented ITS). Overall, Kulik and Fletcher’s (2016) review showed that ITSs can be a very 
effective instructional tool (e.g., the overall average ES in the 50 studies was 0.66). Specifically, 
evaluators’ respective preferences regarding posttest in evaluating the effectiveness of ITSs led 
to discrepancy (Kulik & Fletcher, 2016). For example, average ES on studies with locally 
developed tests was 0.73 while the average ES on studies using standardized tests was 0.13. 
Kulik and Fletcher (2016) suggested that both types of posttest should be used in the ITS 
evaluation studies. As for the condition of the control group used in the studies, the results are 
different for studies with conventional (Median ES is 0.66) and nonconventional control groups 
(Median ES is 0.28). As for the adequacy of ITS implementations, only four studies measured 
implementation adequacy directly. These four studies found that adequately implemented ITSs 
had a stronger effect than inadequately implemented ITSs. Given the findings, Kulik and Fletcher 
(2016) concluded that current ITSs can raise student performance higher than the conventional 
classes, CAI, and human tutors. This affirmative finding encourages developers and researchers 
to keep exploring the affordances of ITSs, as the future ITSs will serve as a substantial component 
in the future eLearning ecosystem. 


Graesser et al. (2018) successfully integrated five distinct ITSs including AutoTutor, Dragoon, 
LearnForm, ASSISTments, and BEETLE-II into a fully functional ElectronixTutor prototype which 
focuses on Apprentice Technician Training (ATT) courses in electronics for Navy trainees. These 
trainees were in the process of A-school training (traditional classroom setting) so that they 
possessed basic knowledge of learning electronics. ElectronixTutor contains ample learning 
materials in their traditional classroom setting (e.g., instructor PowerPoints in ATT) and 
automatically presents specific video lectures based on the diagnosis of student’s performance 


history. Also, the recommender system of ElectronixTutor is an integration of students’ learning 
progression and psychological profile. Student’s performance was scored by two types of 
messages, Completed and Knowledge Component Score (KC score). The ElectronixTutor 


prototype is currently in the process of being assessed and revised. In certain aspects, the 
architecture and functionality of ElectronixTutor are like those of the Generalized Intelligent 
Framework for Tutoring (GIFT). Even though ElectronixTutor is aiming at specific functional 
challenges while GIFT focuses on fostering general, and long-term functionality, improving the 
scalability of ElectronixTutor toward GIFT could also produce benefits to the application of ITSs 
theoretically and practically (Graesser et al., 2018). 


The Generalized Intelligent Framework for Tutoring (GIFT) is ‘a modular, service-oriented 
architecture developed to address authoring, instructional strategies, and analysis constraints 
currently limiting the use and reuse of ITS today” (Sottilare & Holden, 2013, p.1). There are three 
primary objectives within GIFT: tutor authoring, adaptive instructional management, and 
assessment of GIFT effectiveness. In particular, the first objective focuses on providing tools and 
methods (e.g., learner affect modeling, sensor configuration, game-based tutoring, etc.) to make 
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it affordable and easier to build ITSs (Sottilare, Brawner, Goldberg, & Holden, 2012). The second 
objective is aimed at supporting GIFT users to integrate pedagogical models and instructional 
tactics from other systems due to the modularity of GIFT. The last objective emphasized the 
importance of experimental assessment and evaluation of the tools and methods. 


With respect to the last objective of GIFT, Sottilare, Baker, Graesser, and Lester (2018) discussed 
how GIFT, as an experimental tool, can be used to aid Artificial Intelligence in Education (AIED) 
research in three ways. For example, the system can be built with affect sensitivity to 
automatically detect affect in the online learning environment. The most successful example was 
D’Mello et al.’s (2010) Auto Tutor. The system can further develop and assess GIFT models and 
constructs for psychomotor tasks with the help of advanced sensors (Sottilare & LaViola, 2015; 
Sottilare, Hackett, Pike, & LaViola, 2016; Goldberg, Amburn, Ragusa, & Chen, 2017). Finally, the 
system can evaluate and validate team taskwork due to the collaboration characteristic of ITSs, 
while attempting to solve problems or learn knowledge and skills together (Adamson, Dyke, 
Jang, & Rosé, 2014). 


One long-term goal of GIFT is to generalize the authoring of ITSs for taskwork in which the 
learner models, team model, measures of assessment, and interventions are unique to one 
specific domain (Sottilare et al., 2018). 


Affective computing 

With an emphasis on the role of emotions in the study of human-computer interaction, 
researchers are starting to pay more attention to learners’ emotional experiences in the online 
learning environment (Graesser, 2019; Yadegaridehkordi, Noor, Ayub, Affal, & Hussin, 2019). 
The term “Affective Computing” was coined by Picard (1997). Picard (1997) stated that, just like 
human teachers know when and how to provide appropriate support for a student by discerning 
the student’s affective response, a well-designed system should be able to recognize some 
affective states of learners in order to provide personalized feedback and support. The key aim 
of affective computing is to recognize learners’ external affective expressions and connect them 
to their internal emotions. Such recognition of affective states will increase the level of 
personalization learners receive from an affective-aware ITS (Yadegaridehkordi et al., 2019). By 
using affective computing techniques, ITSs can create meaningful and self-relevant responses by 
reacting to learners’ implicit intentions. 


Yadegaridehkordi et al. (2019) reviewed 94 articles on affective computing in education from 
2010-2017 to examine four perspectives of affective computing in the educational domain. Their 
findings include: 


a) The trends in affective computing in education will be mobile devices, such as tablets and 
mobile phones. Therefore, they proposed development of emotional models that can be 
embedded in the current educational systems on the mobile devices. Also, this goal needs a 
collaboration between policymakers, practitioners, and researchers. 


Sixty-Three of the 94 studies reviewed by the authors considered the design of emotion 
recognition and expression systems as their primary research purpose. For example, the 
latest information technology (e.g., cloud computing, green information technology, 
intelligent sensors, cameras, speech prosody, and intonation recognition) and the impacts of 
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color features were considered as the major research direction. In addition, the authors 
pointed out that the use of affective learning in MOOCs, M-learning, and CSCL have not been 
critically explored. 


The integration of textual and visual channels is the most widely used multimodal channel in 
affective computing studies, such as the facial expression method (Lin et al., 2014; Salmeron- 
Majadas, Santos, & Boticario, 2014; Santos, Salmeron-Majadas, & Boticario, 2013; Tjostheim, 
Leister, Schulz, & Larssen, 2015). Also, audio-visual affect recognition was reported as 
another common multimodal channel, which is powerful for capturing and managing users’ 
emotions in a desired system (D'Mello & Kory, 2012; Tao and Tan, 2005). There are also a lot 
of challenges for applying multimodal-based affective recognition systems in practice, such 
as the effectiveness of the integration of different channels, the management of different data 
types, the consensus of the emotional recognition results that come from various methods, 
plus additional challenges. 


Emotional states were discussed under dimensional models (e.g., affective states are 
represented in a multi-dimensional space, such as valence-arousal) and categorical models 
(e.g., emotional states are modeled by discrete emotions such as fear and anger, D'Mello & 
Kory, 2012). The most popular theory/model for describing emotional states is one of the 
dimensional models known as the Control-Value Theory proposed by Pekrun (2006) in which 
students’ beliefs and their value appraisals of the academic environment influence one 
another. However, D'Mello and Kory (2012) pointed out that a mixed classification of 
dimensional and categorical models is needed due to limiting the range of theoretical 
reasons for affective recognition, which might lead to ignoring the other important factors of 
students’ affective states. In addition, the authors indicated that negative emotions (e.g., 
boredom, anger, and anxiety) were considered as the impediments in educational systems 
in most of the studies (Malekzadeh, Mustafa & Lahsasna, 2015; Vogel-Walcutt, Fiorella, 
Carper & Schatz, 2012). The authors suggested that future research should pay more 
attention to specific academic-related demographics in terms of different ages, genders, and 
subject domains to improve the effectiveness of affective computing in educational 
environments. 


Finding ways of triggering positive academic-related emotions and preventing negative ones in 
educational systems are also research directions for the future. D’Mello and Kory (2012) 
provided a guideline to identify the relationship between affective measurement channels and 
special emotional states. For example, facial expressions are used to classify emotions such as 
surprise, fear, anger, and disgust. Textual methods are more often used to investigate boredom, 
anxiety, anger, and enjoyment. The multimodal methods are more suitable for recognizing 
anger, surprise, frustration, disgust, and confusion. 


Calvo and D’Mello (2010) conducted an interdisciplinary review on affect detection in the field of 
Affective Computing (AC) and proposed some important challenges and questions that need to 


be adequately addressed by the AC community. For example, the existing correspondence 


between the experience and the emotional expression is a potentially problematic assumption. 
Some meta-analyses have only yielded small to medium effects (Ekman,1993; Ruch, 1995). Also, 
context is critical for affect detection because context can help clarify the different meanings of 
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the affective expression in different contexts. Further, affect detection must be studied as a social 
process where some affective phenomena are not only understood at the level of individual 
users, but also from the interaction between users and AC applications. Finally, the lack of 
agreement on the performance of affect detection is a major challenge. 


D’Mello and Graesser (2011) investigated the temporal dynamics of students’ cognitive- 
affective states (e.g., confusion, frustration, boredom, engagement/flow, delight, and surprise) 
during deep learning activities on AutoTutor, an intelligent tutoring system with conversational 
dialogue. They found that there is a positive correlation between confusion and deep learning 
(e.g., the persistent confusion showed to be beneficial to learning). This positive correlation also 
be explained by the effect of cognitive disequilibrium caused by confusion. For example, when 
one is confused about a concept, a state of cognitive disequilibrium ensues. However, people 
tend to restore cognitive equilibrium by solving confusion (Graesser, Lu, Olde, Cooper-Pye, & 
Whitten, 2005; Craig, Graesser, Sullins, & Gholson, 2004; Brown & VanLehn, 1980). In addition, 
boredom and frustration have detrimental effects on deep learning (D’Mello & Graesser, 2011). 


D’Mello, Lehman, Pekrun, and Graesser (2014) proposed and tested a theoretical model which 
suggested that confusion which accompanies a state of cognitive disequilibrium can be 
beneficial to learning with proper scaffolding. Such cognitive disequilibrium is triggered by 
negative affective state and events including contradictions, conflicts, anomalies, erroneous 
information, and other discrepant events. These findings explained how confusion influences the 
cognitive disequilibrium process. Specifically, confusion activated a deliberate effort on the 
problem-solving processes. This effort influences knowledge to reorganize and restructure the 
misconception that caused this confusion to correct the existing faulty mental model (D’Mello et 
al., 2014). The bi-directional confusion-frustration transition with an experience of disengaging 
(e.g., frustration to boredom) and annoyance (e.g., boredom to frustration) has been identified. 
Learners need to have the requisite knowledge and skills to resolve the confusion, or 
alternatively appropriate scaffolding to help with the confusion resolution process. D’Mello et 
al.’s (2014) model could potentially benefit the reluctant learners by identifying their confusion 
and increasing their engagement. 


The Application of Personalized Learning and Intelligent Tutoring System 
Augmented Reality (AR) and Virtual Reality (VR). 


AR/VR training systems and Intelligent Tutoring Systems (ITSs) have been widely explored 
separately, but very little work has been done on the integration of ITS and AR (Herbert, Ens, 
Weerasinghe, Billinghurst, & Wigley, 2018). It is well known that both AR/VR and ITSs can 
provide learners with excellent individualized learning experiences and intuitively conveying 
instruction (Herbert et al., 2018; Freina & Ott, 2015). Therefore, we will review the few articles 
that examined the affordances of this combination. 


Herbert et al. (2018) proposed a combination of AR and ITSs in the domain of kinesthetic 
learning and psychomotor learning, Augmented Reality Adaptive Tutors Systems (ARATs). They 
developed a cohesive definition of ARATs which included three components: (a) Use the real- 
world spatial information and dynamically provide feedback using ITSs, (b) use AR to enhance 
learning in real time, and (c) AR-based instruction, create context by using a combination of 
instructional cues and AR. The ARAT conceptual architecture consists of three aspects including 
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the individual learner, augmented environment, and the intelligent tutoring back end. 
Specifically, in the ITS module, intelligent tutoring backend performs the modeling capabilities 
and constantly adapts AR experiences. Then, in the environment module, AR and augmented 
environment dynamically interact with one another and then evolve and adapt over time. Lastly, 
as an individual learner, one’s mental models are developed through the interaction with the 
environment. 


Notably, Herbert et al. (2018) stated that ARATs differ from stand-alone AR training systems for 
the following reasons: (a) ARATs use ITS modeling to support learners’ learning instead of 
simply providing rules, (b) equipped ITS with 3D real-world spatial information can better 
provide modeling support, and (c) AR is used to improve learners’ understanding of instruction, 
rather than displaying the mixing and distracting unrelated content. 


Westerfield, Mitrovic, and Billinghurst (2013) developed and tested an intelligent AR training 
system - Motherboard Assembly Tutor (MAT) by combining AR with ITSs to assist with training 
for manual assembly tasks (e.g., assembling computer motherboard). MAT consists of three 
components including ITS, communication module and AR interface. Specifically, the ITS 
controls the AR interface (e.g., video capture, tracker, and display). The AR interface is blended 
with the student's view of reality via a head-mounted display. The communication module serves 
as a bridge that transfers the important data that is collected from AR interface to the ITS via XML 
encoding. Then, the ITS provides feedback and instruction/guidance after analyzing the 
learner’s data. Westerfield et al. (2013) found that the use of MAT for assembly tasks had a 
significant improvement of the learning outcome over traditional AR approaches. 


LaViola and colleagues (2015) developed a system (e.g., ARWILD system) that combined the 3D 
models and Generalized Intelligent Framework (GIFT-based) tutor to train soldiers. Their 
prototype system successfully moved AR interface from desktop simulations or immersive AR 
systems to the wild and undecorated location that did not have any training infrastructures in 
reality 


Almiyad, Oakden-Rayner, Weerasinghe, and Billinghurst (2017) made use of AR-based 
intelligent tutor systems to assist trainee radiologists to achieve competency in performing 
percutaneous radiology procedures. This system contains three layers. the first layer is the 
intuitive guidance of the depth and angle of the needle during the procedure. The second layer 
is the real-time instructional feedback that was generated from the AR data analysis. Lastly, the 
third layer is a personalized dashboard that consists of the learners’ learning progression and 
performance in multiple relevant metrics (e.g., time-spent on single procedure). They found that 
the effectiveness of this system in improving competence in percutaneous radiology procedures 
is supported because this system is capable of providing feedback on needle angle; 
understanding of the needle angle is the most critical feature that distinguishes learners from 
experts. 


Human-Al Collaboration/shared decision-making 
The Intelligent Web-Based Tutoring System. 


Chen (2007) proposed and tested a genetic-based personalized learning path generation 
scheme in supporting personalized web-based learning. The author found that this proposed 
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learning model is superior to the free browsing learning mode. Reasons for the superiority of the 
personalized learning path are, that the learner receives precise recommendations for 
performance based on system-generated information that continuously modifies the difficulty 
level of the course, which matches the learner’s current competence level (Chen, 2007). 


What is Human-Al Collaboration? 


Artificial Intelligence (AI) systems have become both more powerful and increasingly promising 
in recent years with the development of deep machine learning and hardware (Inkpen, 
Chancellor, De Choudhury, Veale, & Baumer, 2019). However, with the widespread adoption of 
AI systems in real world contexts, researchers and practitioners also raise concerns regarding 
its issues of bias and the difficulty of applying expertise to the decision-making process 
(Dellermann et al., 2019; Kamar, 2016; van den Bosch & Bronkhorst, 2018). Therefore, humans 
are still needed to offer a more holistic perspective in dealing with uncertain and subtle 
decision-making processes. This difficulty ushered in a collaborative era of using AI with human 
interactions. 


In military contexts, van den Bosch and Bronkhorst (2018) explored how humans and AI should 
collaborate to achieve better decision making. They argued that the development of mutual 
understanding in human-Al teams is urgently needed because, like human-human teams, 
human-Al teams have misunderstandings, and the cause of such misunderstanding cannot be 
diagnosed by the machine. To address this issue, van den Bosch and Bronkhorst (2018) 
differentiate three steps of human-Al collaboration that help in mutual understanding. The first 
step is making AI more transparent to the user so that the user can understand how AI produces 
outcomes and what processes are involved (Theodorou, Wortham, & Bryson, 2016). The second 
step involves a bi-directional interaction between human and AI systems. This interaction 
requires that the AI be able to generate query-based explanations based on its understanding of 
the purpose of a human’s request. It also requires that humans allow the Al to provide 
information as it detects misunderstandings, potential bias, or discriminations. The third step 
requires an adaptive collaborative unity in decision making due to the adaptive human-AlI team 
members based on mutual understanding. In this stage, human-AlI collaboration truly harnesses 
each other’s strengths and supplements each other by improving human-system understanding 
using information and feedback during the interaction. 


With respect to the growing fear that AI will soon replace humans in decision making, Jarrahi 


(2018) analyzed why AI systems will only be used for intelligence augmentation, and not become 
a replacement for the following reasons. Human decision making is an intricate process that 
involves intuition and subconscious thought; however, AI can only make decisions based on 
deliberate information gathering and processing. Also, in terms of equivocality in decision 
making that involves conflicting interests of stakeholders, Al systems do not know how to 
integrate emotions, experiences, or contexts of each stakeholder to negotiate and implement 
decisions (e.g., building allies). Therefore, Jarrahi (2018) concluded that human intervention is 
inevitable in a successful human-AlI partnership, and any exclusively Al-based organizational 
decision system is improvident. 


M00Cs 


The definition of Massive Open Online Courses (MOOC) or what qualifies as one has evolved 
over time (Schoenack, 2013). The earlier MOOCs follow the connectivist principle (cCMOOC) 
where many participants self-assemble collections of knowledge, learning activities, and 
curriculum from openly available sources across publicly open platforms (O’Toole, 2013). In this 
perspective, the focus is given on collaborative education, which is achieved through 
knowledge creation, rather than duplicating existing knowledge (Siemens, 2012). MOOCs have 
been redefined in recent years as it is now used as an extension (xMOOC) to access the learning 
activities offered by traditional institutions. This is done through their online platforms (O’Toole, 
2013). This change is based on a behaviorist pedagogical approach and is focused on content 
prepared by universities (Aparicio et al., 2019). However, in a systematic literature review done 
by Liyanagunawardena et al. (2013), they found that there has been no widely accepted list of 
the different types of MOOCs. 


Success in MOOCs 

Success in MOOCs varies based on the actor (e.g., institution, designers, and users) (Klobas, 
2014). One important question about MOOCs is how to measure success. Unfortunately, studies 
modeling MOOCS success, even partially, are scarce (Aparicio et al., 2019). In their literature 
review, Aparicio et al. (2019) found that there are no structural models designed to measure the 
success of MOOCs. One common metric used is the completion rate, which is defined as a ratio 
of enrolled students who satisfied the courses’ criteria to earn a certificate, compared to the total 
number of students who enrolled. Another metric used is the dropout rate, which is 
operationalized as the complement of the completion rate (i.e., 100% - completion rate). 
However, Henderikx, Kreijns, and Kalz (2017) argued that merely looking at course completion 
as a measure for success does not suffice in the context of MOOCs as this measure refers to the 
success of a student and not the success of the MOOC itself. Furthermore, Liyanagunawardena et 
al. (2014) argued that the dropout rate measure fails to identify various forms of dropout such as 
academic failure and voluntary withdrawal. It is important to consider other factors such as 
student’s intention and start date when measuring the success of a MOOC. 


Henderikx et al. (2017) proposed an alternative typology to refine the measurement of success 
and dropout rate in MOOCs. They classified a user as either an inclined actor, disinclined actor, 
or inclined abstainer based on the user’s initial intentions and the subsequent behavior. Those 
who are classified as an inclined actor or declined actor are considered successful. Even though 
the success of the courses should not necessarily entail completion (Pursel, Zhang, Jablokow, 
Choi, & Velegol, 2016), these are indicators where enhancement should take place concerning 
several aspects of MOOCs. Aparicio et al. (2019) found that a gamified learning environment is a 
decisive factor in the success of MOOCs. Klobas (2014) suggested the importance of 
distinguishing a user from a learner in MOOCs as a user can simply register without subsequent 
participation or completion of the course. 


Another approach is to consider the percentage of ‘‘declarative of achievement” with respect to 
the registered population of students who remained active throughout the duration of the course 
(DeBoer, Ho, Stump, & Breslow, 20142014; Jordan, 2014). Simply measuring a student’s success 
ina MOOC by looking at the student’s completion status does not necessarily mean a statement 
of accomplishment (Pursel et al., 2016). For some students, success might be defined as the 


ability to interact with peers who are interested in the same content. Others might define success 
as learning a single concept out of the many ina MOOC. Abbakumov and Van Den (2018) 
proposes to extend the typical measure on how to model student proficiency in MOOCs. This is 
achieved by incorporating non-assessment data, which includes students’ interaction with video 
lectures and practical tasks. The authors proposed cross-classification multilevel logistic 
extensions to the Rasch model, a common Item Response Theory (IRT) model. In their approach, 
they were able to obtain a more accurate model of the student’s proficiency when they 
incorporated the student’s behavior. 


Student Engagement in MOOCs. According to Kopp (2011), the following are the conditions that 
encourage the involvement and engagement of learners in Connectivist MOOCs (cCMOOCs). This 
includes (1) the social presence of the facilitators and participants, (2) feeling competent and 
confident in using the different tools, (3) learning in an autonomous fashion without the provision 
of organized guidance by facilitators, and (4) the emergence of critical literacies such as 
collaboration, creativity, and a flexible mindset, which is a prerequisite for active learning ina 
changing and complex learning environment. 


How students engage in MOOCs varies according to the needs of the learner (Mawas, Gilliot, 
Garlatti, Euler, & Pascual, 2018). In the survey done by Mawas et al. (2018), they found the 
following reasons why students engage in MOOCs; finding a new job, getting a promotion, 
meeting family expectations, earning a higher salary, solving a specific problem (accounts for 
the motivation for the 60% of the courses), and help to pass a class. 


Motivation is identified as an important contribution to student engagement in a MOOC 
(Milligan, Littlejohn, & Margaryan, 2013). Furthermore, Salmon, Pechenkina, Chase, and Ross 
(2017) argue that the motivation of students in MOOC is mostly intrinsic. Shrader, Wu, Owens, 
and Santa Ana (2016) surveyed the participants who were registered in different courses to know 
the reasons or their motivations for taking MOOCs. Interestingly, only a few participants (3.3%) 
registered to gain a course certificate, while the majority wanted to either broaden their 
knowledge (65.6%) or were curious or generally interested in the topic (35.6%). This finding is 
in consonance with Barak, Watted, and Haick (2016) where they found students who participate 
in a MOOC appear not to pursue a certification, but they are merely interested in the learning of 
the MOOC content. Catenazzi, Sommaruga, de Angelis, and Gabbianelli (2018) suggested that 
participants consider interactivity as the key factor to improve motivation and engagement. 
Nawrot and Doucet (2014) that MOOCs should implement less time-consuming assessment 
methods. 


The type of students who are mostly involved in MOOCs have high levels of self-regulation. 
Learners who are working as professionals in a field relevant to the MOOC content and those 
students working towards a higher education degree have higher self-regulation levels (Hood, 
Littlejohn, & Milligan 2015; Kizilcec, Pérez-Sanagustin, & Maldonado, 2017). Shrader et al. (2016) 
found that participants who hold a masters or a Ph.D. were twice as likely to complete the course. 


Issues and challenges 

Students whose confidence levels are low are believed to not take up connectivist learning 
(Kopp, 2011). In fact, low confidence often becomes a barrier to these students. One problem 
being faced by MOOCs is retention, as MOOCs typically have a high number of students 


enrolled but suffer from low engagement (Ventista, 2018) and high attrition rates (Koutropoulos 
et al., 2012). According to Jordan (2014), the average completion rate of MOOCs they surveyed 
was only 5%. It is interesting to note that it is still not well understood what the factors or the 
learning components are which support student retention in MOOCs (Fournier & Kop, 2015). 


Among the 12 MOOCs analyzed by Aleman de, Sancho-Vinuesa, and Gomez Zermefio (2015), an 
atypical course had a completion rate of 22.53%, which was relatively higher than the average in 
the literature. They identified the possible factors that may have contributed to this. One factor 
was the careful process of course design and the included technological resources (e.g., 
animated readings and interactive exercises). Another was the use of practical tools (e.g., 
various Google+ Tools) along with the different communication strategies which were 
implemented by the teaching staff. The goal was to motivate participants to continue engaging 
with the contents of the MOOC and answer the exercises. 


Graham (2006) identified several issues with MOOC, as with other self-directed environments. 
These included: less immediate feedback and guidance, lack of personalization, a high tendency 
of students to procrastinate, and becoming overwhelmed by the resources made available. Mak, 
Williams, and Mackness (2010) suggested that there has been unacceptable behavior (e.g., 
forceful intellectual debates, feelings of participation being demanded, and rude behavior) from 
some MOOC participants, which has led other participants to cease posting on forums. 


Issues with Peer Feedback in MO0Cs. 

Peer feedback in MOOCs, especially in Coursera, has often been criticized due to its anonymity 
and lack of checking for plagiarism (McEwen, 2013), inconsistent with a lack of feedback on the 
peer-assessment itself (Watters, 2012), poor comprehension of peers in terms of understanding 
the feedback given to them by students who provided the feedback and spent significant effort 
(Kulkarni, Wei, & Le, 2013), and the question on its trustworthiness by the students (Floratos, 
Guasch, & Espasa, 2015). Li et al. (2016) suggested that peer assessment is more accurate when 
students participate in the creation of the rating criteria. However, in the context of MOOCs, 
these rating criteria are just provided to the students. Another issue on peer feedback is the 
presence of patriotic bias (Kulkarni et al., 2013), where students tend to give higher marks to 
peers who are from the same country. 


Jordan (2015) examined several MOOCs in terms of completion rate. Those courses that use 
peer grading or a combination of peer and auto have a completion rate of less than 10%. Those 
that used only autograding had a completion rate of more than 20%. This could be a possible 
solution to the high attrition rate. Unfortunately, not all courses can utilize autograding (i.e., 
based on domain). As mentioned above, Nawrot and Doucet (2014) recommended less time- 
consuming assessment methods, which further strengthens the case against peer assessments as 
these activities are time-consuming. 


mesoM00C: A new framework for MO0C. 

A mesoMOOC is a framework that challenges current and future MOOC designers to embed 
principles which are known to be effective in reaching adult learners (Schoenack, 2013). This 
includes considering the orientation process, embedding a connectivist synchronous component 
to the class, providing online formative and summative assessment, and developing subsections 
within the class. 


Current trends in personalizing MO0CS. 

To reduce the drop-out rate in MOOCs, there has been a growing trend of research in MOOC 
personalization and adaptation to improve user’s engagement (Sunar, Abdullah, White, & Davis, 
2015). Mawas et al. (2018) suggested that the ‘‘one-size-fits-all” policy is not relevant in MOOCs. 
In their study, they identified some key elements for their content personalization approach in 
the context of lifelong learning. They identified these elements by examining different projects 
related to personalized MOOCs. They identified the following elements, which are grouped into 
three levels, namely: the learning level (learning goals, learning experience, and learning 
recognition), the visualization level (learning path), and the content level (content granularity). 
In another study, Yu, Miao, Leung, and White (2017) offered a perspective on how advances in 
artificial intelligence can be leveraged to enhance learning in MOOCs. This includes how 
knowledge representation tools can enable students to adjust the sequence of learning to fit their 
own needs, how optimization techniques can efficiently match community teaching assistants to 
MOOC mediation tasks to offer personal attention to learners, and how virtual learning 
companions with human traits such as curiosity and emotions can enhance learning experience 
on a large scale. 


Affect in MO0Cs. 

Dillon et al. (2016) collected affect data in an introductory Statistics MOOC using self-reported 
surveys, namely The Self-Assessment Manikin and a discrete emotion list. They found the 
following emotions commonly reported: hope, enjoyment, and contentment. This study has been 
the first in the literature to attempt to conduct such methodology. 


Social Learning and Engagement through Social Media 


Designing learning through student collaboration and engagement is a current focus of many 
educators at all levels (Martin, Martin, & Feldstein, 2017). To meet the challenge of engaging 


students, social learning through social media has emerged as a readily available pedagogical 
tool (Martin et al., 2017). Educators are looking to social media sites (e.g. Facebook), 
microblogging sites (e.g. Twitter) and other educational learning platforms (e.g. Yellowdig) to 
engage students (Martin et al., 2017). Bingham and Conner (2015) describe social learning as 
walking a path that begins with what the learner wants to learn, making a commitment to 
learning in front of others, then sharing what has been learned. They add that social learning has 
evolved from a simple focus on available learning tools into a combination of using online 
learning tools and institutional culture shifts to encourage knowledge transfer and personal 
connections. These fluid learning opportunities are meant to simultaneously foster learning and 
increase enjoyment in learning. At its core, social learning is collaborating with others to make 
sense of information and to create new ideas in much the same way that colleagues and friends 
have been interacting since the beginning of time, except that the colleague may be thousands 
of miles away and the friends may never meet except in the social space. The technology tools 
and social platforms are the means to an end, connecting and collaborating with others for a 
specific purpose (Bingham & Conner, 2015). 


Bingham and Conner (2015) also aid in the definition of what social learning is by highlighting 
what it is not. Social learning is not only for those involved in knowledge making or 
development, what Bingham calls ‘knowledge workers’, but is for people in all types of pursuits. 
It is not a substitute for formal education or employee development, nor is it synonymous for e- 
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learning or informal learning since neither of those environments are necessarily social in 
nature. Further, it is not a MOOC, although MOOCs may use social learning as a tool in their 
learning scheme. Finally, social learning is not a new method of searching for information that is 
found on social sites, since the searcher may not be contributing to the knowledge. Social 
learning takes advantage of the social nature of all humans, who shape their realities by 
scaffolding prior knowledge with new information and experiences. 


Social Media 

People learn throughout their lives and formal learning environments only constitute one 
method by which learning occurs (Venter, 2019). With today’s technological advancements, 
learners have access to many tools that facilitate informal learning opportunities through 
collaboration, such as Social Networking Sites (SNSs) (Venter, 2019). Additionally, informal 
learning opportunities are often self-initiated interactions that happen during mandatory 
interactions on formal Learning Management Systems (LMSs) containing course content (Venter, 
2019). In either case, these informal learning tools allow students the opportunity to source, edit, 
share, track, and monitor their individual learning activities, and to follow others’ activities 
throughout their collaboration (Venter, 2019). Romero-Hall (2017) maintains that social media 
spaces are a form of informal online learning that is frequently used due to its popularity, ease of 
use, and global accessibility. Cook, Pachler, and Bradley (2008) viewed mobile learning ona 
continuum between informal and formal learning and suggest that learner-centered scaffolding 
by a tutor (via texts, for example) could aid in bridging the gap from informal to formal learning, 
in certain situations. 


SNSs attract millions of users and serve to connect strangers with shared interests, views, or 
activities or are created to be a gathering place for groups of people with similar interests or 
needs (Boyd & Ellison, 2007). College and university students find social media sites a popular 
place for engagement. For example, by 2007, the popular SNS, Facebook, already had a strong 
following on college campuses (Ellison, Steinfield, & Lampe, 2007). Additionally, 62% of 18-29- 
year olds reported using Instagram and 67% of the same age group reported using Snapchat 
(Perrin & Anderson, 2019). Ellison and colleagues (20007) also found that students reported 
spending 10 to 30 minutes per day on Facebook and have a friend base of 150 to 200 people on 
their Facebook profiles. Sites differ in their information and communication tools such as 
blogging ability, mobile connectivity, and photo or video-sharing (Boyd & Ellison, 2007). These 
SNSs allow individuals to create public or semi-public profiles in the systems such that users are 
enabled to articulate and reveal their network of associations. 


The role of communication in knowledge generation 
As previously stated, learning is not done in isolation but is, simultaneously, both an individual 


and social process (Brown, Collins, & Duguid, 1989). Online learning provides students with the 
opportunity to interact with one another despite differences in location, time, or background; 
however to reap the benefits of actual learning, mutually beneficial interactions between 
learners and other learners or instructors, must be established, nurtured, and reciprocated 
through shared feelings of purpose and trust (Venter, 2019). 


It is well established that online learners can struggle with disconnectedness and feelings of 
isolation in online courses (Venter, 2019). If online learning isolates, and deep learning is a 
social activity, then some bridge must be constructed to bring online learners to the social 


connectedness that aids learning (Venter, 2019). Some may see social media as that bridge. 
Social media technologies have revolutionized the way people connect and interact, both 
personally and professionally (Chugh & Ruhi, 2018). 


Sharples, Taylor, and Vavoula (2006) state that communication is the driver in learning. 
Discourse assists learners in three dimensions, namely the cognitive, social, and interactive 
levels (Xing & Gao, 2018). Conversations with others allow learners to become cognitively active 
by asking questions, relating experiences or knowledge, elaborating on content, and 
interpreting findings. Communication also helps learning through sharing perspectives, which 
increases our understanding of others. 


Shea, Fredericksen, Pickett, Pelz, and Swan (2001) agreed that online learning should be viewed 
through a social lens. Online learning through communication has an intimate association with 
social presence, which is the ability of a participant to represent themselves online ina 
community (Rourke, Anderson, Garrison, & Archer, 1999). Social presence is linked to 
perceived learning (Rovai, 2002), and is demonstrative, dynamic, and cumulative (Kerhwald, 
2010). For example, it is demonstrative because learners must be able and willing to reveal 
themselves to others (being present without making responses is called lurking and does not 
qualify as social presence). Social presence is dynamic because students’ social presence is 
fluid and is altered over time, depending on the number, quality, and frequency of interactions 
with their online cohort. Finally, social presence is cumulative over time as students gradually 
reveal themselves so that other students get a sense of their relationship to one another 
(Kerhwald, 2010). In addition, Rybas (2008) also suggested that communication in online 
communities should not be graded against face-to-face communication, but it should be 
appreciated as unique and acknowledged as possessing attributes that set it apart from other 
communication forms found in other learning environments. 


Social presence in online communities is relegated to the subjective and is affected by 
everyone’s perspectives and immediate point of view, in addition to their accumulated 
experiences. Individuals also subjectively choose the frequency and level of social interaction in 
the online community (Kerhwald, 2010). Therefore, instructors cannot assume individuals will 
bring the same level of social experiences to the community or that the social relationships are 
valued similarly or equally between participants (Oztok, Zingaro, Makos, Brett, & Hewitt, 2015). 


Social Capital Theory 
The idea of social capital has been used by sociologists to study human relationships and 
connections (Oztok et al., 2015). Social capital in an online space is the relationships that are 


formed in a social network and how those relationships facilitate action (Coleman, 1990). In an 
educational context, social capital is the intangible aspect of relationships that exist within the 
family, the institution, and the community in the form of obligations or expectations that serve to 
aid or hinder academic success (Ho, 2019). Stodd and Reitz (2018) suggest that in social learning 
situations, social authority is collectively granted based upon trust, reputation, fairness, and 
investment made over time. This social authority, along with social capital are the determining 
factors in a student’s ability to collaborate and learn in social settings. 


The social capital theory is a common framework to investigate students’ motivation to share 
knowledge (Diep, Cocquyt, Zhu, & Vanwing, 2016). Specifically, Chiu, Hsu, and Wang (2006) 


commented that social capital has many dimensions such as social interaction ties, trust, 
reciprocity, identification, shared vision, and shared language. Differences in culture (for 
example, Eastern versus Western culture) can influence cultural communication patterns. Diep 
et al. (2016) and Ho (2019) stated that social capital is a valuable theoretical construct to study 
student performance disparities between nations. 


Two types of social capital that are studied in online learning research are Bridging Social 
Capital and Bonding Social Capital (Putnam, 2001). Bridging social capital is depicted as 
inclusive and encompasses ways to bring diverse people together, while bonding social capital 
strives for exclusivity and allows people of like interest to be united (Putnam, 2001). Oztok et al. 
(2015) state that bridging social capital can account for how online learners form a learning 
community from a combination of their online social interactions and social presence (online 
persona). Bridging social capital begins when students connect and interact with community 
members from different walks of life and serves to begin some form of relationship between 
previously unacquainted students (Venter, 2019). Venter (2019) states that, as diverse student 
populations interact, structural “holes” can develop which must be bridged by “brokers” whose 
job is to facilitate acquaintance so that ties between students can begin to form. If the bridging 
does not complete, disconnected students fall into these holes and fail to get exposure to new 
knowledge or ideas (Venter, 2019). 


After the learners meet in the online community, the goal is to move from a bridging sort of 
social capital to a bonding form that helps unify groups around their common interest in the 
course (Oztok, et al., 2015). These close ties develop when learners find others that share their 
own characteristics or similarities (Venter, 2019). This may result in certain community members 
coming to value collaboration and critique from other members of the community, for the 
purpose of learning from this more trusted or respected group member (Venter, 2019). 
However, it is important to remember that encouraging student communities to become 
interconnected is no guarantee that social capital will eventually develop (Venter, 2019). 


Venter (2019) noted that development of social capital may be impacted by student access to 
required technologies, socio-economic history, and educational background and skill set. 
Furthermore, learning communities are not static, coherent, nor homogeneous since students are 
virtual colleagues with no history at the beginning of an online course (Oztok et al., 2015; Venter, 
2019). Social capital has been positively linked to educational attainment, educational 
achievement, and psychosocial factors, but understanding exactly how social capital is related to 
achievement remains unclear (Dika & Singh, 2002). 


Relationship Between Social Presence and Social Capital 

Chiu et al. (2006) stated that the biggest challenge in distance learning environments is fostering 
an online community where people are willing to share their views and experiences. The 
concepts of social presence and social capital are interrelated and highly correlated (Oztok et 
al.,2015). It is foundational to understand that different relationships within communities or 
networks hold different perceived value for each person in the group (Dika & Singh, 2002; Oztok 
et al., 2015). Oztok et al. (2015) called for shifting our understanding of social presence to 
include working within communities, rather than viewing social presence as an individual 
characteristic of a learner. 


Oztok et al. (2015) found that bridging and bonding types of social capital both have significant 
relationships to social presence. Bridging social capital’s impact on social presence was larger 
and stronger than that of bonding social capital. It is noteworthy that strong close ties between 
students and weak distributed ties are both influencers of social presence, however, 
communication between weak distributed ties is more closely related to social presence. They 
suggested that this phenomenon (weak ties being more closely related to social presence) may 
be due to students using bridging social capital in online interactions, because online learning 
practices naturally foster weaker and more diverse relationships in required online 
communications. Furthermore, some students may not desire closer ties with the other online 
learners, preferring instead to keep communication shallow and broad with certain online 
course participants. Social presence has implications for the formation of a robust online 
community of learners and further research is needed to help elucidate the relationships 
between social capital and social presence. 


Relationship Between Social Capital and Social Media 

The internet has been linked to increases and decreases in social capital (Ellison, Steinfield, 
Lampe, 2007). Nie (2001) concludes that internet use decreases interpersonal interactivity 
communication. While Bargh and McKenna (2004) found the contrary to be true, that the internet 
fosters relationships and is not a threat to community life. 


Venter (2019) investigated a diverse group of open distance learning institutions and found that 
students engage in both formal and informal collaborative learning activities in relation to the 
online classes. It was also posited that the use of formal and informal learning provides 
participating students with Personalized Learning Environments (PLEs) which can moderate both 
the strong and weak ties of developing social capital in the online community. Students in online 
learning environments experience different dimensions in which they can leverage social capital 
(Venter, 2019). These dimensions are that social interactions foster structural, relational, and 
cognitive opportunities for collaboration, resource sharing, and experience sharing (Cummings, 
Heek, & Huysman, 2006; Venter, 2019). Research findings have emphasized the importance of 
online interactions for forming the weak ties of bridging social capital (Ellison et al., 2007). In an 
early study, Ellison et al. (2007) found that internet usage alone did not predict social capital, but 
intensive Facebook use did. Both bridging and bonding social capital accumulation varied 
based on the degree of the students’ self-esteem and their satisfaction with life, demonstrating 
that students with low self-esteem and low satisfaction with life made gains in social capital 
through Facebook use. This prompted Ellison et al. (2007) to surmise that intensive Facebook 
use could be helpful to the group of students with low self-esteem and low satisfaction with life. 


Uses of Social Media in Online Learning 

Venter (2019) found that students look elsewhere for learning tools when the formal learning 
does not provide enough information to satisfy their needs. Venter (2019) found that a popular 
social media tool, WhatsApp, was a useful tool for gaining “just-in-time” information and for 
helping students understand assignment expectations by sharing ideas and insights. 
Aleksandrovai and Parusheva (2019) found that students had different patterns of utilization of 
social networking sites, depending upon the purpose of the interaction. Students in 
Aleksandrovai and Parusheva’s (2019) study used Facebook for content sharing or 
communication with colleagues. However, wikis and LMSs were the preferred tools for content 
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creation and additional learning. Students expressed that, as they used the social media sites, 
they first assessed the quality of the posting before interacting with the writer (Venter, 2019). 


Students who venture into the social spaces for learning are demonstrating student agency 
through self-regulation (Venter, 2019). As students seek to manage and choose options for 
learning and participate in collaborative activities that provide social capital benefits and 
engage in opportunities of learning. Venter (2019) found that informal collaborative activities 
exceeded the mandatory levels of engagement from LMS interactions required by course 
instructors. Some students in the study sought out study groups before engaging in an online 
learning experience, and the students’ commitment to the group was that of a “family” of 
learners which was continued throughout the time the student was enrolled in the degree- 
seeking program of study. Boyd and Ellison (2008) stated that SNSs generally support pre- 
existing social relations. Generally, students in a study by Ellison, Steinfield, and Lampe (2007) 
study reported spending significantly more Facebook time with their offline connections than 
with other Facebook friends. 


Facilitation and Collaboration 

Social media seems ideally suited to work with constructivist learning theories that encourage 
and incorporate student engagement as part of sense-making and learning (Marek & Skrabut, 
2017). According to Currie and colleagues (2014) Some educators feel that social media’s power 
should be harnessed for use in the classroom, and state that instructors would be remiss to 
ignore social media as a learning tool. They claimed that since students already feel ownership 
of the social media environment, that perception improves communication, problem solving, 
and genuine student reflection. Furthermore, social media usage fosters professionalism, trust, 
and respect between students and between students and instructors. A study by Moran, Seaman, 
Tinti-Kane, and the Babson Research Group (2011) reported that nearly two-thirds of faculty had 
used social media during a class session. They found that 30% had posted content on social 
media sites for students to read or view outside of class, 40% had required students to use social 
media sites as part of a class assignment, and 20% had required social media postings for a 
course (Moran et al., 2011). 


Chugh and Ruhi (2018) cited that Facebook offers multiple benefits for learners such as 
increased teacher-student and student-student opportunities for interactions and engagement, 
improved performance, and the convenience of learning on demand. These findings have been 
supported by Northey, Bucic, Chylinski, and Govind (2015). Currie et al. (2014) viewed 
Facebook as a tool that enhances interactions in authentic environments and believe the platform 
can reinforce the “hidden curriculum” of accountability and professionalism in a way that other 
reflective activities cannot. Students not only receive instant feedback about their own reflective 
posts, but they must also consider their comments from the standpoint of how those views are 
perceived by others (Currie et al., 2014). A Turkish study of prospective English language 
teachers found that Facebook usage in language study helped students be more reflective of 
their teaching and increasing their metacognitive awareness (Balcikanli, 2015). Furthermore, this 
study found that student teachers also realized benefits in learning to use online technologies. 


Social media has been used by educators for more than facilitating collaboration or 
engagement, social media has been used for the delivery of content and teaching materials, 
educational information, and class updates (Chugh & Ruhi, 2018; Hamid, Chang, & Kurnia, 2009). 
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Cheston, Fleckinger, and Chisolm (2013) performed a systematic review of the use of social 
media for medical education and found the included studies demonstrated favorable results in 
learner satisfaction, attitudes, knowledge, and skills, although the authors noted that well- 
designed studies were rare. Romero-Hall (2017) found that in a study of graduate students that 
while students used social media sites for personal reasons, they were not always connected to 
the social media sites their program of study provided. However, those graduate students that 
did use the course-provided SNS reported benefits such as an improved sense of belonging to a 
group of professionals, ability to interact with others in distant locations, and networking 
opportunities for career advancement (Romero-Hall, 2017). Chromey, Duchsherer, Pruett, and 
Vareberg (2016) found that students in higher educational classrooms were willing to use social 
media for class purposes if certain criteria were met, namely that the use of social media was 
deemed appropriate. Appropriate use required that it be convenient to use, was the best tool 
available to use in the circumstances, use was voluntary, and no personally identifying 
information was required to participate (Chromey et al., 2016). Kobayashi (2017) found that 
students did not find social networking as valuable as other forms of asynchronous media. 
Roblyer, McDaniel, Webb, Herman, and Witty (2008) uncovered that students are more likely 
than faculty to use Facebook to support learning. 


Currie et al. (2014) acknowledged that, as with most innovations, there are benefits and risks 
associated with the use of social media sites. Some benefits, according to Currie et al. (2014) 
include the following: 


1) Engagement can be richer due to improved communication, over formal university 
forums. 

2) Students already pursue social media interaction, and institutions can easily meet their 
students in the social sites or struggle to make connections with students in other ways. 

3) Social media sites can help students engage in authentic reflection and self-evaluate their 
own perspectives, while learning about professionalism in their career by watching and 
listening to their peer groups’ perspectives. 


According to Currie et al. (2014), risks of social media site usage in education include the 
following: 


1) Some students can have difficulty keeping distinct lines between personal and 
professional domains in their lives. Negative interactions that would not be allowed on 
institution-controlled forums are less restricted on social sites. 

2) Social media creates an availability expectation on instructors’ time that is outside of 
stated response times and students may use social sites to impatiently press faculty 
members when their contact expectations are not realized. 

8) Institutions have little enforceable control over social media sites, which also frustrates 
accountability measures for protecting processes and people. 

4) Privacy issues can arise because social media sites can quickly transmit personal or 
private information to members of a supposedly closed group. Groups can experience 
bias and negative conversations without barriers. Social media allows unchecked 
viewpoints and personalities to be reflected and shared with others. 

Faculty or staff who establish academic sites bear responsibility for students that may 
require the site builder to take action to ensure student safety, professionalism, and 


suitability for jobs. When students exhibit disturbing behaviors or make questionable 
comments on social sites, faculty are put in a dilemma about revealing such information 
to the authorities who can intervene. 


A recent study of the effects of social media use on academic performance uncovered that using 
social media in academic pursuits was not a relevant indicator of academic performance as 
measured by cumulative grade point average, but using social media for non-academic 
purposes, especially gaming or multitasking, showed academic performance effects (Lau, 2017). 
Both gaming and multitasking were significant and negatively predictive of academic 
performance. Ravizza, Hambrick, and Fenn (2014) found that higher rates of internet use for non- 
academic purposes during classroom time was associated with lower examination scores for 
students. Furthermore, test score variances were apparent regardless of academic ability, 
demonstrating that students are not efficient multi-taskers when engaging in in-class internet 
activities. 


For any benefit to be derived from the use of SNSs, students and faculty must be willing to accept 
the technology into their academic lives (Choi & Chung, 2013). This willingness has been studied 
in the context of the Theory of Reasoned Action (TRA) as an extension of the TRA called the 
Technology Acceptance Model (TAM) (Choi & Chung, 2013). The TRA is a framework that takes 
beliefs and motivation for a behavior and connects them to attitudes and norms of behavior 
which culminates in an actual behavior (Choi & Chung, 2013; Davis, Bagozzi, & Warshaw, 1989), 
while the TAM states that the decision to use technology is based upon both the perceived 
usefulness and the ease of use (Davis et al., 1989). In applying the TAM to SNSs, the perceptions 
of usefulness and ease of use had robust effects on a person’s decision to engage in social 
media; the subjective norm of behavior (would people important to me believe I should or 
should not participate in the action) and perceived social capital were significant predictors of 
the perceived usefulness and ease of use. Individual differences in subjects, such as gender, 
age, and race were not significant to the intention of using technology (Choi & Chung, 2013). 


Digital Citizenship 

Students using social media may perceive social networks as beneficial to work, private, and 
educational relationships; however, what happens online stays online forever (Fineman, 2014). 
Digital citizenship, which is the norms of behavior in relationship to technology use, has its 
beginnings in computer ethics (Ribble, Bailey, & Ross, 2004; Xu, Yang, MacLeod, & Zhu, 2019). 
Digital citizenship has nine components: online etiquette, communication, education, access, 
commerce, responsibility, rights, safety, and security (Ribble et al., 2004). College students’ 
social media use has been related to multiple violations of digital citizenship, such as plagiarism, 
online information disclosure, fraudulent activities (Xu et al., 2019), cyberbullying (Watts, 
Wagner, Valasquez, & Behrens, 2017), and internet addiction (Salehan & Hegahban, 2013). It is 
necessary to evaluate the use of social media as a learning tool considering these digital 
citizenship concerns (Xu et al., 2019). Instructors and students must use careful judgement when 
using social sites to ensure that students possess adequate social media competency (Xu et al., 
2019). Social media competency involves six constructs, namely, social media self-efficacy, 
social media experience, effort expectancy, performance expectancy, facilitating conditions, 
and social influence (Alber et al., 2015). 


Privacy Concerns 

Privacy concerns for SNSs are well known (Such & Criado, 2018). One definition of privacy 
postulated by Westin (1967) and revisited by Margulis (2011) is that privacy is the claim of 
institutions, individuals, and groups to self-determine who, how, and to what extent others 
should be granted knowledge about the institution, individual, or group. SNSs are continually 
adding features that assist people who want to form networks for a variety of purposes, such as 
sharing information or interacting with others (Chen, 2018). Three prominent concerns of the 
privacy of SNSs are data collection, data control, and third-party sharing (Mahmoodi et al., 2018). 


Due to their data collection backdrop, SNSs contain vast repositories of personal information 
(Wu, 2019). Accessing features of SNSs on apps on mobile technologies has led to increasing 
concerns that identifiable information is aggregated, archived, and stored across different media 
platforms (Wu, 2019). Wu’s (2019) research demonstrates that a person’s need for self-identity is 
positively associated with their privacy management behavior patterns, which results in 
increased self-disclosure in SNSs. Wu (2019) also stated that the anticipation of social capital can 
be a driving force in a person’s electing to self-disclose. Wu (2019) calls the current environment 
of valuing privacy but indulging in self-exposure on SNSs a ‘privacy paradox’. Trepte and 
Reinecke (2013) found that there was a reciprocal relationship between SNS usage and self- 
disclosure, people who were willing to self-disclose were more likely to use SNSs and the use of 
SNSs made people more willing to self-disclose. These effects were moderated by the amount of 
social capital that SNS users received by participating in social media content generation (Trepte 
& Reinecke, 2013). A fundamental privacy issue is then, how can privacy be protected while 
simultaneously encouraging self-disclosure? Powell, Wimmer, Rebman, and Abdul al (2019) 
remind institutions that, while social media use is nearly ubiquitous for millenials, the internet 
platforms are subject to abuse of data and security risks and Diaz, Golas, and Gautsch (2010) 


warn that requiring students to participate in a social media sites may be subject to FERPA 


guidelines. 


Protecting social media users’ privacy can help prevent disastrous cybercrimes and illegal use 
of data obtained through breaches in social media sites, examples of these include 
cyberbullying, phishing scams, identity theft, and cyberstalking (Such & Criado, 2018). Many 
privacy threats fall under a privacy inference attack which means that public information on 
social media accounts, such as demographics (e.g., age, gender, major, likes, dislikes) are used 
to infer attributes about the user. The party obtaining the information may range from data 
brokers, service providers, advertisers, or cyber criminals (Beigi & Liu, 2018). These inferences 
can be friend-based attacks, behavior-based attacks, or a combination of the two (Beigi & Liu, 
2018). Inherent in social media use however, is not just damage to the privacy of the individual 
posting the information, but to the privacy of other people pictured in or associated with the post 
(Such & Criado, 2018). The aspect of privacy for all associates for Multi-person Privacy (MP) in 
posts (Such & Criado, 2018). The current mechanisms of dealing with MP is through 
tagging/untagging and reporting inappropriate content, which may be too little too late as the 
poster has initial control of the post, and while photos may be removed by tagged parties, 
message content may remain (Such, 2018). Liang, Liu, Lu, and Wong (2018) found that even when 
photos are deleted, a significant lag time exists until the content is truly unavailable. 


Deletion Delay for Different Social Media Sites 


From Liang et al., 2018 


Platform Days until Deletion Occurs 


Facebook 


Twitter Immediately 


Instagram 


MySpace, Tumblr 


Flickr 14 


Digital identities may be altered if students are reluctant to post their true opinions in class 
assignments, which Powell et al. (2019) suggest may affect them professionally, privately, or 
educationally. Powell et al. (2019) found that content posted in five out of seven social media 
sites was found in a Google search when the posters had turned off the data privacy options. Ina 
study by Powell et al. (2019), results indicated that Facebook, Twitter, YouTube, Screencasts, 
Prezi, Voice Thread, and LinkedIn all had privacy settings available for users, although enabled 
privacy settings were not the default setting in any of the applications. Only Screencasts data was 
not easily found by a Google search and all SNSs were deemed a security risk if the SNS use was 
required for class purposes (Powell et al., 2019). 


Security 
Data collection is a necessary part of higher education institutions, and the collected data 


contains sensitive information such as names, addresses, social security numbers, test scores, 
behavioral assessments, and personal health information (prnewswire.com, 2018). Additionally, 
research institutions hold valuable intellectual property, which is potentially put at risk in 
internet and social media use (prnewswire.com, 2018). A recent study of 17 U.S. industries 
revealed that the educational system ranks last for cybersecurity risks (prnewswire.com, 2018). 


Social media security can be approached from two perspectives, identification of the risks and 
attempts to mitigate the risks (Beigi & Liu, 2018). Some risks that can leave a university system 
vulnerable are app security, which can leave metrics and testing information unprotected, 
patching cadence, which if delayed or slow, can open systems up to vulnerability, and network 
security, which needs continuous monitoring since the use of cloud services can become 
vulnerable at any time (prnewswire.com, 2018). Recommended security necessities are to move 
security measures to the end point of the system, which is anywhere data ‘lives’ or is accessed 
(Stevens, 2019). Social media use can be difficult for technology security teams, as they have 
long relied on perimeter provisions like firewalls and secure web gateways. Furthermore, these 
teams are reluctant to require students and faculty to download cybersecurity solutions onto 


devices (Stevens, 2019). One solution is to screen individual devices for security threats to 
determine their ‘health’ before allowing access to university resources (Stevens, 2019). This 
approach is part of a zero-trust model where constant surveillance of the devices and system 
entry points is maintained (Stevens, 2019). 


User-generated data is vulnerable from two different types of attacks, identity exposure and 
attribute exposure (Beigi & Liu, 2018). Beigi and Liu (2018) stated that SNSs should anonymize 
user-generated data prior to publishing it. User-generated data is susceptible to being traced, 
which renders the users vulnerable to fraud, violence, or exposure of sensitive information such 
as location (Mahmud, Nichols, & Drews, 2014), age (Nguyen, Smith, & Rose, 2011), and 
relationships of trust or distrust (Beigi, Tang, & Liu, 2016; Beigi & Liu, 2018). Stevens (2019) states 
that the avenues to privacy in SNS&s are to limit actual self-disclosures by limiting frequency and 
content of postings, or by being more selective before friending (Ellison, Vitak, Steinfield, Gray, 
& Lamp, 2011). 


Potential concerns about the incorporation of soctal media not discussed in the Iiterature 

There is relatively limited research published about the use of social media in distributed 
education to facilitate the learning process. Accordingly, it seems pertinent to outline a few 
concerns that the authors of this report have considered, but not empirically tested, based on the 
presumption that they may influence the effectiveness of social media in distributed learning 
contexts. Relevant work is cited when relevant work could be located. 


Providing information to students through social media when the students also use an LMS 
platform for the purposes of their typical instruction potentially splits the information between 
two distinct platforms; one presumably at least somewhat controlled by the institution (the LMS, 
although this is often cloud-based hosting provided by the contracting company) and one 
presumably not controlled by the institution (the social media site). This is potentially 
problematic as it seems plausible that the information provided through the LMS may be viewed 
as official documentation by students, whereas information provided through social media may 
not. Additionally, it would potentially require students to check two different repositories to find 
the information necessary for their course. 


Another concern has to do with the ownership and security of typical LMS solutions compared to 
social media in relation to student assignments, grades, and potentially course content. Due to 
global variations in regulating student privacy, little has been written about privacy in academic 
settings (Marek & Skrabut, 2017). However, the U.S. has strict regulations in place under the 
Family Educational Rights Privacy Act (FERPA) which prohibits academic institutions from 
releasing personally identifiable information about students, except regarding a limited 
directory (Marek & Skrabut, 2017). FERPA prevents institutions from sharing information about a 
student's grades, schoolwork, or behavior to anyone without the student’s permission (Watters, 
2011). FERPA compliance, though rigidly stressed by institutions, gives little remediation for 
individuals who have had privacy violations, since these students cannot sue the school, but can 
only report the violation to the U.S. Department of Education (Marek & Skrabut, 2017). In line 
with FERPA, students’ educational records have certain protections regarding their privacy. 
Accordingly, some educationally relevant communications with students through non-secure 
means, such as social media, potentially poses FERPA compliance issues (see Diaz et al., 2010). 


Especially regarding the Department of Defense (DoD) training, some content may be classified 
and not for general consumption. Adding a non-DoD controlled social media component to a 
course could potentially open avenues for security violations, due to the participants discussing 
topics that should not be discussed outside of those with the appropriate security clearances. It 
seems plausible that simply posting otherwise innocuous content on a social media site, even if 
the group is marked as ‘private’ or the joining of the group is somehow controlled, could create 
security issues due to the machine learning algorithms the site may use collecting and analyzing 
the users posts or communications. 


Another potential concern of incorporating social media into learning is whether the information 
on social sites is accurate and if students can choose reputable information (Kammerer, Brand- 
Gower, & Jarodzka, 2018). Studies have shown that people are beginning to make queries on 
SNSs rather than using a search engine for information (Morris, Teevan, & Panovich, 2010). 
Harper, Moy, and Konstan (2009) report that although Q & A sites have launched search engine 
companies to expand their body of knowledge, and simultaneously engage users, the social Q & 
A sites have failed to gain momentum, except in Asia. Furthermore, these social Q &A sites 
failed to produce a reliable source of information for users (Harper et al., 2009). Kammerer et al. 
(2018) stated that intentional and unintentional misinformation is rampant in SNS and Salmeron, 
Macedo-Rouet, and Rouet (2016) found that users rely more on answers from self-declared 
experts than non-experts when alternative messages are provided on the site. Salmeron et al. 
(2016) identified that children value expert status in the same way as adults do; however, adults 
place more significance on source information in Q &A sites. Prior research has demonstrated 
that users bring their own bias, opinions, and attitudes on a topic which can heavily influence 
their information retrieval pathway (Kammerer t al., 2018). Zejda (2010) noted that trust matters 
in the social networking sites, and that one common abuser tactic is to establish trust with their 
targets prior to acting fraudulently. Moorehead et al. (2013), ina meta-analysis examining the 
role of social media in the healthcare domain, stated that there are benefits to social media use 
for health information communication, and that it is critical to validate the information obtained 
as reliable and of appropriate quality. Furthermore, Moorehead et al. (2013) reiterated some of 
the privacy and security concerns mentioned previously in this present work. 


One additional concern that should be explored when contemplating the use of social media in 
learning is the possible misrepresentation of oneself in the online world (Hongladarom, 2016). 
Hongladarom (2016) posits that the conundrum of social media profiles is that they may 
represent an actual person or may represent who that person would like to be. Furthermore, the 
online self may be a fabrication meant to deceive. Yang and Brown (2016) stated that self- 
presentation is a form of self-disclosure that is strategic in the amount of information a user 
presents, the intimacy level of the presentation, the positivity of the information, the authenticity 
of the information, and the intentionality of the disclosure. Chester and O’Hara (2007) found that 
students in their study generally perceived that they operated on a desire to be honest, but 
many chose pseudonyms and images that were not their actual names and faces. Students were 
mostly satisfied with their self-presentation choice, but more than half of them said that, in 
hindsight, they would have chosen their self-presentation differently (Chester et al., 2007). 
Grades were related to the self-presentation choices made by the students, as those who used 
real names and images scored the highest in assessments (Chester et al., 2007). Impersonation 


plays a major role in successful social engineering scams on social media sites and credibility of 
the posters must be examined to prevent harm (Algarni, Xu, & Chan, 2017). 


Blended Learning and Social Media 

Some instructors value social media interactions for teaching with active learning and for 
introducing the students to possible uses of this technology for future learning and as a 
professional tool (Megele, 2015). Megele (2015) redesigned a blended course with a social 
media component (used for learning, assessment, and student engagement) and found that the 
learning outcomes of the module were facilitated and enhanced. Specifically, Megele (2015) 
found that students experienced an improvement in their understanding of personal learning 
networks and e-professionalism. Forbes (2017) remarked that it is necessary to prepare 
students, who will be professionals in their fields, in the proper use of social media by helping 
them understand the dynamics of a social presence and learning networks. Bodell and Hook 
(2014) found that occupational therapy students perceived an improvement in their confidence 
levels of using social platforms for professional networking after participating in a blended 
networking class designed to assist students in navigating social sites for networking and 
professional development. 


Pak and Verbeke (2013) used social media to extend learning in a studio-based blended 
learning course. Students felt the SNSs were convenient for learning while instructors enjoyed 
the ability of the SNSs to represent design information in novel ways and to add to 
communication forms for the class. They were able to use collaborative mapping to facilitate a 
collective construction of students’ memory of urban spaces which helped students understand 
the project site, learn from experts, and engage in peer learning. This indicated that a student’s 
participation in the SNS was related to their progress up to a certain point. However, the 
investigators were unable to elicit a cause for this relationship due to the correlative nature of 
the study. 


One example of the use of social media in a blended learning environment in a first-year 
experience course (McCarthy, 2010). The major finding was an increase in interaction between 
local and international students in the SNS, which was attributed to the international students 
having more time to process and respond to postings. Because of the increased online 
interactions, the dialogue between the student groups became more interactive and prolong 
during in-class meetings. 


Mobile learning trends 


What is the definition of mobile learning? 

Kamilali and Sofianopoulou (2015) described mobile learning (or m-learning) as a method for 
learning on the move, which is limited in time and device use. Sharples et al. (2007) further 
defined mobile learning as “the processes of coming to know through conversations across 
multiple contexts among people and personal interactive technologies” (p. 225). Kamilali and 
Sofianopoulou (2015) noted that mobile learning is not simply a presentation or shrinking of 
existing e-learning materials in mobile devices. Rather, it should be designed such that it is able 
to link people in real and virtual worlds, create learning communities between people on the 
move, provide expertise on-demand, and support a lifetime of learning. 


The aviation industry has been one of the early adopters of mobile learning, for example, tablet 
computers are used as electronic flight bags (Kearns, 2013). How mobile devices are used in this 
industry can be considered as just-in-time training or performance support. In this approach, 
knowledge or skill is not explicitly intended to be retained by the learners. Mobile devices are 
simply used to help in performing a task at a given moment (Murray et al., 2014). 


Characteristics of mobile learning 

Frohberg et al.’s (2009) state-of-the-art report found that, even though mobile devices are 
designed to be communication tools, communication and social interaction played a small role in 
the mobile learning projects they analyzed. Furthermore, Ally (2009) identified learner-centric, 
personalized instruction, spontaneous, portable, and situated as characteristics of mobile 
learning. In a systematic literature review done by Imtinan et al. (2013), the authors were able to 
identify the common and popular characteristics of mobile learning as suggested by researchers 
in the field. This includes usability, collaboration, context, control, connectivity, mobility, 
content, blending, technical support, and cost. Kearns (2013) identified some of the features of 
mobile devices that can be taken advantage of to revolutionize the teaching practices in the 
context of aviation training. This includes push notifications, location-specific applications 
through Global Positioning Systems (GPSs), massive storage at a low weight, and video/camera 
functions. 


In blended learning, Horton (2011) suggested a “sandwich” strategy which places classroom 
instruction in between the e-learning or m-learning elements. This strategy was adopted by 
Kearns (2010) in the context of aviation training. The pre-training is meant to deliver the 
foundational knowledge and skills in preparation for the classroom instruction. The classroom 
instruction is for topics which require human interaction or specialized equipment. Lastly, the on- 
the-line training extends the training to the workplace, which is beyond the classroom. Both pre- 
training and on-the-line training can be supported by m-learning. 


Kearns (2013) acknowledges the challenge of bringing back professionals to classrooms ona 
regular basis, particularly in the aviation industry. To address this, snap-courses were 
suggested. These courses were designed to be completed over a longer period and should 
facilitate distributed practices. The following are the recommended characteristics of these 
courses: duration has to be approximately 5 minutes, must include interaction, must be designed 
to facilitate intrinsic motivation, facilitate discussion among learners, incorporate repetition to 
promote retention, integrate quizzes that facilitate retrieval practice, should allow learners to 
choose a convenient time to complete training, and encourage learners to complete training 
over a longer period of time. 


Advantages of mobile learning 

Through mobile learning, employees can solve problems via handheld devices in situ. The cost 
of these devices is relatively low, which allows for the delivery and creation of multimedia 
content. This makes informational resources more accessible, enabling continuous and situated 
learning support. This can lead to a reduction in training costs (Elias, 2011; Rudestam & 
Schoenholtz-Read, 2009). Additionally, student motivation can be improved by supporting their 
basic needs of autonomy, competence, and relatedness (Nikou & Economides, 2018). 


Effectiveness of mobile learning 

Studies on mobile learning mainly use survey questionnaires to solicit students’ perspectives 
(Wu et al., 2012). Students from elementary and higher education settings reported that they 
were able to learn via this medium (Thornton & Houser, 2005; Lu, 2008; Al-Fahad, 2009). Ina 
literature review conducted by Wu et al. (2012), it was found that mobile devices and PDAs are 
mostly used. Furthermore, 86% of the 164 mobile learning studies surveyed by Wu et al. (2012) 
were found to have positive outcomes. 


Dbiquitous Jearning 

Ogata et al. (2009) define ubiquitous learning (u-learning) as an everyday learning environment 
that is supported by mobile and embedded computers and wireless networks in our daily lives. 
It has been examined in the workplace, formal, and lifelong learning settings (Pimmer et al., 
2016). Most research is in the domain of language and linguistics, health sciences, and computer 
sciences. Mobile and ubiquitous learning are strongly interconnected concepts. They are 
conceived as tools that allow learners to access information regardless of their physical context. 
They can also provide learners with location-based information. 


Characteristics of ubiquitous Jearning 

Collaboration plays an important role in these u-learning environments (Yang, 2006; Hwang et 
al., 2011). Huang et al. (2011) surveyed the literature and summarized the characteristics of u- 
learning. First, it should be able to meet the user’s spontaneous learning needs, encouraging 
learners to take the initiative in knowledge acquisition. It should be able to support constructivist 
and self-regulated learning. It must be based about instructional activity and is context aware. 
Also, interactivity is important in the learning process and the context needs to provide a 
community of learners. Finally, it should be adaptive to the learner and must be personalized. 


Evaluation in ubiquitous learning 

Huang et al. (2011) proposed an evaluation method framework for ubiquitous learning 
environments which is based on meaningful learning. It was composed of four stages: (1) the u- 
learning practice, (2) a macro view, (3) a micro view, and (4) u-learning refinement. After a 
learning activity is carried out, learners are asked to evaluate it using a meaningful learning 
questionnaire scale. The entire group is analyzed both from the macro and micro view to assess 
the degree of meaningful learning achieved in the u-learning environment. 


The macro view involves the use of a five-point Likert rating scale questionnaire which covers 
five dimensions, namely, was the learning active, authentic, constructive, cooperative, and 
personalized. Each dimension contains three items, which are statements on what a learner 
could do within a learning environment or with learning activities. The questionnaire used for the 
macro view, covers five dimensions with three statements for each dimension. 


The micro view uses a questionnaire based on the analytic hierarchy process (AHP). This 
questionnaire can determine the pros and cons associated with multiform criteria through paired 
comparisons. Questions are about the u-learning activities. 


Despite this framework, our literature search revealed that there is limited research on the 
effectiveness of ubiquitous learning. 


Future of Mobile Learning 

The ability to learn from a mobile device is still a relatively new frontier for distributed learning, 
as such this is an area that would benefit from an outline of criteria. Mayer (2019) has suggested 
the following six criteria points: objective measurement of learning outcomes, focus on 
instructional methods specifically directed at mobile technology, designed experiments over 
observational studies, remain neutral about the possible value of mobile technology in learning, 
incorporate relevant theories of learning and motivational research, and to use mobile 
technology as a research tool. This is not to say that these criteria points have not been absent 


from research. There have been many designed experiments (Melby-Lervag, Redick, & Hulme, 


2016), work regarding motivational and learning theories (Renniger & Hidi, 2016), and others 
have moved forward with using mobile technology for research purposes (Xie, Heddy, & 
Vongkulluksn, 2019). These criteria should be used as a constructive means of how to go about 
taming the wild west that is the mobile learning platform. To fully understand and answer the 
currently pressing questions; how do mobile devices affect the process of learning, how does 
mobile learning allow new opportunities for influencing the learning process and outcomes, and 
how mobile technology allow for previously uncollectable data (Bernacki, Greene, & Crompton, 
2020). 


Mobile Blended Learning 

Mobile blended learning is the use of mobile devices in conjunction with other technologies 
used for learning (Suartama & Setyosari, 2019). Mobile internet technology has created 
opportunities for blended learning (Suartama et al., 2019). Mobile learning, as well as 
connecting formal to informal learning, helps improve student participation, achievement, and 
learning (Suartama et al., 2019). 


Suartama et al. (2019) stated that mobile blended learning requires thoughtful and systematic 
design. To determine if mobile blended learning is a good content fit, Suartama et al. (2019) 
suggested that designers conduct a three-phase pre-analysis of the design problem by 
considering the learners’ prior knowledge and characteristics, a learning object identification to 
qualify what knowledge should be taught about the subject, and an analysis of the blended 
learning environment. After the pre-analysis phase, designers should extend their designs to 
determine learning activities and resources and determine how assessment will be conducted 
(Suartama et al., 2019). 


Since blended mobile learning is fashioned well for informal learning, Hou et al. (2014) found 
that college students a blended mobile interface could provide for an improved focus ona 
museum's on-site exhibits and a mobile learning platform. Additionally, Hou et al. (2014) found 
that the blended mobile learning may increase the interaction of students between the on-site 
exhibits and the learning website which may help interaction with the museum’s learning 
activities. 


ESL classrooms are often plagued by too little classroom time to accomplish lesson objectives 
that will assist students in becoming fluent in English Jamal, 2015). Jamal (2015) recommended a 
‘learning by doing’ approach in a mobile blended environment. Jamal (2015) asserted that these 
approaches can increase student autonomy and self-directedness. Avci and Adiguzel (2017) 
concurred and added that the Mobile-Blended Collaborative Learning model has been used in 
and out of the classroom to give students authentic and collaborative opportunities to practice 
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English language learning in a project-based approach. Using blended-mobile collaborative 
learning demonstrated that students practicing in authentic situations for real purposes 
improved their vocabulary and communication skills (Avci et al., 2017). Further, authentic 
practice improved the students’ recognition of colloquial English and adding instant messaging 
improved the quality of their work and had positive effects on their performance (Avci et al., 
2017). 


Microlearning 

Microlearning has been defined as a new learning approach which is based on small learning 
units and short-term focused activities (Hug et al., 2006; Lindner, 2007). However, Hug (as cited 
in Eibl, 2007) argued that it is not a well-designed paradigm. It is therefore better to focus on its 
features and characteristics rather than on its definition (Eibl, 2007). 


Effectiveness of microlearning 

When microlearning was used as a strategy to teach a class, the students were found to have 
better learning than the traditional group, and an enhanced self-perceived autonomy 
(Mohammed et al., 2018; Nikou & Economides, 2018). However, these studies had participants 
from elementary and high school levels. 


The relationship between microlearning and mobile learning. Mobile-based microlearning is 
considered a relatively new approach that combines features of mobile learning and 
microlearning through the delivery of small learning units and short-term learning activities 
through mobile devices (Hug, Lindner, & Bruck, 2006). Furthermore, mobile-based 
microlearning is personalizable, adaptive, ubiquitous, and context-aware (Bruck, Motiwalla, & 
Foerster, 2012). It has been an emergent practice in corporate training and workplace learning 
(Clark et al., 2018; Goggins, et al., 2013). It can be used together for the development of short 
online activities in MOOCs that can be embodied in everyday life (Kamilali & Sofianopoulou, 
2013). Jahnke et al. (2019) conducted a literature review and found that these microlessons have 
an average length of not more than five minutes. In the same study, they also conducted a series 
of interviews with industry leaders. They found that industry leaders would prefer to have such 
lessons to be shorter (30 to 90 seconds). 


In their paper, Nikou and Economides (2018) argued that mobile-based microlearning has been 
considered a successful learning strategy in the workplace (Bruck et al., 2012; Werkle et al., 
2015) and that it improves both the learning performance and motivation in professional and 
corporate working environments (Munoz-Organero et al., 2012; Pimmer & Pachler, 2014; Wen & 
Zhang, 2015). 


Design challenges of mobile microlearning. Jahnke et al. (2019) summarized some of the 
design challenges of mobile microlearning. First, there is too much information being presented 
on small screens. Another is the absence of clear contact information (e.g., instructor). The use of 
smartphone devices may distract the learner. Finally, issues such as accessibility, technical 
issues, and affordability were raised. 


Design challenges in microlearning platforms. In their paper, Jahnke et al. (2019) unpacked 
inherent design principles and decoded characteristics of existing mobile microlearning 
platforms that are targeted for outside learners and those in traditional offices. These were 


obtained after triangulating the results from the three data sets of academic literature, industry 
reports, and interviews with industry professionals. Similar principles were grouped together, 
and eight major themes emerged. The following table summarizes their findings: 


Interactive micro-content for closing Interactive content 
practical skill gaps 


Practical problem-solving 


Chunked courses Snackable, not crammed single 


Instructional flow of activity-based model of Instructional flow, sequenced, engaging 
instruction 
Rich of diversity of media formats 
Instant feedback 
System design App availability 


Push notifications 
Track learning progress 


Browsable, independent, searchable micro- 
lessons 


Curriculum provides single lessons; may Embedded into a broader curriculum 
sum up into certificates/degrees 


These principles were used to evaluate existing platforms by developing a simple heuristic (yes 
or no for each principle). However, Jahnke et al. (2019) noted that further research may provide 
more detailed categories beyond the simple heuristic they developed. 


Issues in microlearning. 


Jahnke et al. (2019) identified some of the issues with the mobile microlearning design. One is 
the absence of reflection. It also addresses learning topics and outcomes that are easy to learn. It 
does not include higher order thinking skills of Bloom’s taxonomy. Another is that it follows the 
behaviorist approach (i.e., learning by clicking and not by creating artifacts). Finally, they argue 
that this design is geared towards automation where answers are already known. 


Mobile learning and the different theores of learning. 

Naismith et al. (2004) were able to identify 6 categories of learning activities in their literature 
review. These were taken from an activity-centered perspective. The categories are behaviorist, 
constructivist, situated, collaborative, informal and lifelong, and learning and teaching support. 
The authors provided examples of the use of mobile technology for each. Using these 
categories, Pimmer et al. (2016) conducted a systematic review of empirical studies of mobile 
and ubiquitous learning in higher education settings. Their analysis focused mainly on 
instructionist, situated, constructionist, and a hybrid of situated, constructionist, and 
collaborative. They found that positive outcomes were mainly associated with the instructionist 
and hybrid designs. The instructionist benefits were due to frequent learning activities, while the 
hybridization links formal education with informal and personalized learning. The authors 
acknowledged that there is limited evidence to legitimize the broad application of mobile 
learning in higher education. 


Micro and Blended Learning 
Our review did not locate any studies that discussed microlearning specifically regarding 
blended learning. 


Video 
Video Use 


Distance learning can occur in a variety of ways, but the fastest growing learning is the 
use of asynchronous video (Malaga & Koppel, 2017). Video can be used for the delivery of 
course content, such as a lecture, or can contain supplemental information for student learning 
(Malaga & Koppel, 2017). For the purposes of this discussion video(s) refers to instructional 
videos which are multimedia productions for the purpose of helping people learn targeted 
material (Fiorella & Mayer, 2018). Watching videos is commonplace today and students are 
familiar with a variety of hosting sites such as YouTube and Vimeo (Malaga & Koppel, 2017). 
Miner and Stefaniak (2018) demonstrated that both students and instructors believe that video is 
an appropriate way to communicate course content while Scagnoli, Choo, and Tian (2019) 
affirmed that video lectures are considered an effective means of delivering content and of 
providing the necessary teaching presence in a virtual learning environment. Scagnoli et al. 
(2019) were able to associate students’ positive perceptions of video learning with positive 
overall learning experience ratings and with perceived impact on learning. Furthermore, video 


instruction was able to enhance students’ perception of engagement, due to the impacts of 
learner control and teaching presence. 


Kay’s (2012) comprehensive review of the literature found that in addition to the affective 
and cognitive perceptions of students toward video learning and improved learner control, 
students found benefits in improved study habits and in their learning performance with video 
use. MacHardy and Pardos (2015) added that unhelpful videos do not add to student 
performance and suggest instructors should vet videos before assuming the inclusion of videos 
will enhance student learning. For example, research clearly demonstrates that students learn 
better from videos that follow research-based principles of effective design, because the 
principles enable learners to cope with the new material in ways that respect human cognitive 
capabilities (deKoning, Hoogerheide, & Boucheix, 2018). 


To maximize the benefits of video learning, it is necessary to understand the constructs 
under which the video learning is most effective. Brame (2016) suggests that video content be 
designed with consideration given to cognitive load, student engagement, and active learning. 
For a discussion of cognitive load in multimedia learning, see the cognitive load theory section 
of this document. Mayer (2014) describes a framework, called social agency theory, which posits 
that social cues affect deep learning. According to the social agency principle, multimedia 
materials can be designed with social cues, which stimulates a student’s motivational 
commitment to begin and maintain active cognitive processing. Students who experience an 
activation of a social response within themselves find that this social response facilitates an 
increase in active cognitive processing and an increase in the quality of the learning outcome. 
When the instructional message lacks appropriate social cues, there is no activation of a social 
response and no increase in active cognitive processing or improvement in the quality of the 
learning outcome. The appropriate social cues found in the video presentation are beneficial to 
enable students to respond to another social being and commit to learning. These social cues 
contrast with realism cues, which would result from a feeling of physical presence, which do not 
necessarily affect learning. 


Video Design 

Generally, in video design, Brame (2016) suggests that cognitive load can be 
appropriately tolerated by students when the material in the video is segmented or chunked into 
smaller segments, students are signaled to notice important information, the modality principle 
is followed to make audio and visual content match, and weeding is used to eliminate extraneous 
information. Ibraham (2012) states that video learning faces three main challenges, namely, the 
transience of the information on the screen, a lack of focused attention due to a lack of guidance 
on the important aspects of the message, and the incorporation of extraneous content that 
distracts the learner by taking up working memory space. The problem caused by transience 
can be overcome by using segmentation, while the lack of directed attention can be alleviated 
with the use of signaling (Brame, 2016; Ibraham, 2012). Finally, the problems associated with 
extraneous content can be minimized through weeding (Ibraham, 2012). 


Segmenting refers to breaking multimedia works into smaller, meaningful pieces so that 
the learner can exercise control over when to continue with the presentation (Fiorella & Mayer, 
2018). Segmenting can help learners control essential processing (a concept analogous to 
cognitive processing), which involves learners selecting relevant words and images in a 


multimedia presentation and organizing them for understanding (Mayer & Pilegard, 2014). 
Students experience essential overload when encountering materials presenting critical content, 
depending on their prior knowledge; however, instructors cannot eliminate the essential 
elements of the presentation. Therefore, dividing the lesson (segmenting) into meaningful 
chunks can help students with processing demands. They added that segmenting could have a 
more significant role for learners if the material is complicated, unfamiliar to students, or 
presented at a fast pace. Another cognitive load reducing strategy is the use of pre-training to 
improve a student’s prior knowledge with unfamiliar materials. Fiorella & Mayer (2018) agreed 
that segmentation is important for controlling essential processing overload and adds that 
learner control is pivotal to improving learning outcomes when using video materials. Biard, 
Cojean, and Jamet (2018) found that learner control alone (interactive format) was not sufficient 
to improve procedure learning by students, as students rarely interrupt the video. Lowe (2004) 
found that novice learners did not employ learner control as effectively because of lacking 
direction in discerning important information. Instead, Biard et al. (2018) found learner 
interactive systems, with additional system-controlled interruptions that occur after the 
presentation of pivotal learning segments, showed superior student learning. Segmented 
instructional videos reinforce procedural representations for novice learners and reduce 
cognitive load. Schnotz and Rasch (2005) found that learners with high prior knowledge had a 
high enabling function when allowed to manipulate animations, while learners with low prior 
knowledge were enabled by simulations without manipulation options. Wouters, Tabbers, and 
Paas (2007) suggested content designers follow a social cognitive model of sequential skill 
acquisition in which learners progress from merely observing skills to becoming independent 
and self-regulated performers. 


Two current areas under investigation are the importance of the onscreen presence of 
the instructor and whether video instruction should occur in the first or third person for improved 
interaction. These concepts are part of an increasing body of work that investigates how social 
cues can prime social responses in learners that result in deep cognitive processing and 
improved test performance. Social cues include the personalization principle, voice principle, 
image principle, and embodiment principle (Mayer, 2014). To maximize student engagement, 
Brame (2016) suggests improvements in learning can be made if the video is brief, uses 
conversational language, audio is spoken enthusiastically, and the videos are inserted into 
curriculum at prime moments when the material will be most relevant. Mayer (2014) agrees that 
conversational language usage (personalization) helps students learn more deeply than a more 
formal verbal style. Conversely, Schworm and Stiller (2012) did not find any difference in 
retention outcomes between highly personalized or weakly personalized presentations; 
however, personalized presentations improved transfer knowledge. The personalization 
principle operates with boundary effects such that high achieving students and long lessons may 
negate the benefits of personalization (Mayer, 2014). Domain-specific prior knowledge showed a 
reversal effect with personalized video presentations (Stiller & Jedlicka, 2010). In lower 
knowledge learners, personalization has shown positive effects in drawing, labeling, structural 
knowledge, and transfer (Stiller & Jedlicka, 2010). In higher-knowledge learners, drawing and 
labeling was improved; however, structural knowledge was not impacted, and transfer 
performance was reduced (Stiller & Jedlicka, 2010). 


Student learning is also impacted by the voice principle, which suggests that people 
learn more deeply when multimedia is presented with a human voice compared to a machine 
voice (Mayer, 2014). Designing audio clips with native speakers using a standard accent 
conveys a level of social presence for the learner and makes them feel as if they are being 
directly spoken to during the content delivery. The voice cues may affect the level of social 
response a learner engages in with the content (Mayer, 2014). However, research has shown 
mixed results in relation to the voice principle (Chiou, Schroeder, & Craig, 2020; Craig & 
Schroeder, 2017, 2019; Santally & Goorah, 2012). For example, Santally and Goorah (2012) found 
that, while students preferred a natural voice in audio narration, there was no significant 
difference in learning gains with natural voice narration over synthetic audio use. Craig and 
Schroeder (2017, 2019) suggested that the voice effect may have been due to the technologies 
used in early studies, as more recent work has shown that there have been largely no 
differences in learning outcomes between videos narrated by modern machine voices and 
recorded human voices. 


The image principle is that for deep learning to occur, people do not need to see the 
speaker’s image on the screen during the presentation (Mayer, 2014; van Wermeskerken, 
Ravensbergen, & van Gog, 2018). The image principle states that the social response benefits of 
showing the instructor on the screen during a video is counteracted by the extra cognitive 
processing that accompanies the instructor’s presence (Kizilcek, Bailenson, & Gomez, 2015). 
One large study (n = 2,951), in which students could exercise choice on viewing videos with or 
without an instructor on-screen, uncovered that students who saw the instructor’s face perceived 
that they had a more pleasant learning experience; however, 35% of students decided against 
viewing the videos showing the instructor’s face for self-reported reasons including avoiding 
distraction (Kizilcek et al., 2015). To avoid such distractions, Kizilcek et al. (2015) designed 
videos in which students had strategic instructor placement to maintain teacher presence while 
reducing distractions. These authors found that the image principle was supported as learning 
did not change regardless of instructor presence (Kizilcek et al., 2015). Furthermore, attrition 
rates were not altered in either the constant or strategic instructor placement conditions (Kizilcek 
et al., 2015). Kulh and Zander (2017) found that the personalization principle was reversed when 
the subject matter contained adverse content. 


The embodiment principle is that people learn more deeply when the on-screen agents 
use human gesturing, eye contact, movement, and facial expressions (Mayer, 2014). Lusk and 
Atkinson (2007) found that college students taught with a fully embodied agent (locomotion, 
gaze, and gesturing) produced more accurate answers at near and far transfer. A meta-analysis 
by Schroeder, Adesope, and Gilbert (2013) found a significant, although small (g = 0.19), 
positive effect of pedagogical agents on learning. Further, Schroeder et al. (2013) found that 
learning was more easily facilitated when students utilized on-screen text rather than narration, 
and the usefulness of pedagogical agents was greater for K-12 learners than for post-secondary 
students. Li, Wang, Mayer, and Liu (2019) found that pedagogical agents that used specific 
gesturing improved students’ ability to pay attention to task-related elements of the material and 
performed better on retention and transfer tests. The embodiment principle also has boundary 
conditions in that when a negative social cue is used on screen, the embodiment principle may 
be negated (Mayer, 2014). 


Fiorella & Mayer (2018) suggested learning outcomes improve when instructional videos are 
filmed from a mixed perspective, which use both the first-person (student) and third person 
(instructor) perspective. First-person perspective may assist student engagement by helping the 
learning experience be seen from the student’s perspective (Fiorella & Mayer, 2018). One 
relevant consideration for engagement in videos is modeling (Hoogerheide, van 
Wermeskerken, Loyens, & van Gog, 2016). Hoogerheide et al. (2016) found that for secondary 
education students, adult models were better for student learning than peer models if the target 
material is considered by students to be more appropriately known by adults. A study by 
Hoogerheide, Loyens, and van Gog (2016) showed that the gender of the model or observer had 
no effect on learning or near transfer. Hoogerheide, Loyens, and van Gog (2014) found that 
university students’ performance was affected by study intention (test versus explanation of the 
content), but students who were required to make a webcam video (actually explain the content) 
experienced a significant effect on fostering transfer. Hoogerheide, Renkl, Fiorella, Paas, and 
van Gog (2019) found that students who had recorded a video to teach peers outperformed 
students who only studied the example, demonstrating that teaching on video is a successful 
learning strategy for students. Further, Hoogerheide, Deikjers, Loyes, Heijltjes, and van Gog 
(2016) found that explaining on video but not by writing aided learning more than restudy. Using 
teaching video production as a homework assignment improves test performance compared to 
re-study or summarizing (Hoogerheide, Visee, Lochner, & van Gog, 2019). Besides modeling, 
demonstration videos can improve motivation and task performance for users (van der Meij, 
2017). Combining demonstration videos with review videos resulted in an additional 
improvement in task performance (van der Meij, 2017). 


Betrancourt and Benetos (2018) asserted that video is best delivered with consideration 
to other aspects of the production such as camera angle, instructor presence, and external 
design features. In relation to design features, van der Miej, Rensink, and van der Miej (2018) 
found that demonstration-based training for children was not improved by practice before or 
after video software training. Fiorella & Mayer (2018) acknowledged that students use videos to 
watch procedures and build mental models from information, but that watching alone will be 
incomplete for learning. Practice without feedback has not been shown to be effective for 
learning; however, feedback on practice attempts allows students the opportunity to adjust and 
correct their knowledge as learning progresses (Fiorella & Mayer, 2018). Kapur (2016) stated 
that performance does not always equate to learning. Kapur (2016) called for instructional 
designs to aim for understanding the nature of the learners’ prior knowledge and to take 
advantage of productive failures in unguided problem solving to build upon that prior 
knowledge base. Productive failures can demonstrate what students already understand and can 
be used to engage students and build upon prior information levels (Kapur, 2016). Betrancourt 
and Benetos (2018) cautioned that the role of practice should be evaluated in adults before 
generalizing that practice is of no benefit when used with video training. 


To assist with active learning, Brame (2016) encouraged the use of questions in videos, 
whether interactive- or guiding-type, to stimulate thinking. Brame (2016) also suggested using 
the video material as part of a larger assignment. Wouters et al. (2007) called for a four- 
component instructional design model (4C/ID) that uses multiple cognitive processes to aid 
learning. These include elaboration and induction, which allow learners to construct accurate 
mental models, and compilation and strengthening, which allow learners to make these models 
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automatic (Wouters et al., 2007). Constructing schemas or models can be accomplished by 
cueing, pacing, prediction, learner control over the appearance of information, working in pairs 
to take turns observing and performing tasks, utilizing reflection prompts, and personalized task 
selection (Wouters et al., 2007). Personalized task selection is important for automaticity of 
learner schemas (Wouters et al., 2007). 


Video Lectures and MO00Cs 

Video lectures have become a vital learning component to the structure of MOOCs 
(Stohr, Stathakarou, Mueller, Nifakos, & McGrath, 2019). Their ability to helpfully convey the 
information to the learner has been shown to be successful not only across demographics such 
as age, but also successful regardless of the learners’ specialization, meaning non-experts can 
benefit just as much from the video format in MOOCs (Stohr et al., 2019). The specifics of video 
creation for MOOCs have been examined, from how the lecturer is portrayed, to what length of 
video is most successful in viewer retention (Luo, Zhou, Li, & Xiao, 2018). Video usage still has 
some challenges in the MOOC environment, as things like language barriers are not as easily 
translatable as a text format (Valor Mird6, Baquero-Arnal, Civera, Turr6, & Juan, 2018). However 
even these obstacles are being overcome with the use of automatic speech recognition (ASR) 
and machine translation (MT), the use of which saves anywhere from 25-75% of time that would 
normally be used in translations (Valor Mir6 et al., 2018). As the use of video lectures in MOOCs 
continue to grow in the University settings, more research is being conducted regarding 
meeting the needs of the different culture’s MOOCs expand into (Bayeck & Yvonne, 2018). 
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Virtual reality /Augmented reality / Simulations 
Virtual Reality and Simulations 


The term “virtual reality” has become a sweeping term in conversations regarding the 
newest form of reality technology. Virtual Reality or “VR” has come to mean anything from 
immersive 3D world environments to 2D overlays on the world using icons or a Heads-Up 
Display (HUD) interface (Hepperle, Weif, Siess, & Wolfel, 2019). Thankfully, the term 
“simulation” is more straightforward, referring to any imitation of a situation or process to 
produce, either through a computer or other means, the feeling of an experience without 
actually undergoing that experience (Simulation, 2019). While the term simulation is a much 
broader term that can encompass a wide variety of topics covered in this review, the term virtual 
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reality can potentially be misused or misunderstood when talking about the topics of different 
forms of reality; e.g. some refer to virtual reality when talking about a computer that displays 3D 
models with zoom, rotation, and virtual movement (Miinzer & Zadeh, 2016), while some use the 
term to describe virtual worlds that multiple users can explore through the use of headsets and 
walking controls (Nelson & Ketelhut, 2007). While both fall under the spectrum of VR, this review 
will define and explain the subgroups contained in VR to enable more in-depth analysis of VR 
research going forward. 


Virtual Reality, Augmented Reality, and Mixed Reality 


Virtual Reality. The term VR can refer to a wide variety of applications, including everything 
from online game environments in which the user interacts through means of a keyboard to an 
avatar (Kim, Park, & Baek 2009), to the fully immersed environments utilizing headsets to display 
full virtual worlds (Nelson & Ketelhut, 2007). As this definition is quite broad and unwieldy, this 
review will further break down the different VR experiences available. There are many 
differences in types of virtual reality, that are defined by the equipment used (Ritz & Buss, 2016), 
the type of virtual environment (Peterson, 2010), and how a user can explore the virtual 
environment (Jungwon, Jangwoo, & Jeha, 2010). However, there are also two large sub-groups of 
virtual reality that also need to be defined, augmented reality and mixed reality. 


Augmented Reality. Augmented Reality or AR refers to extending or enhancing the real 
environment with a digital overlay of graphics and/or sounds in real time (Siegle, 2019). AR has 
become increasingly prominent in the lives of the average individual due to its inclusion in 
phone applications in recent years. AR has been involved in many phone and tablet applications, 
either through the use of placing pre-created 3-D objects into the live video image or AR that is 
activated by the device finding a trigger image to manipulate (Siegle, 2019). An example of the 
pre-created 3-D object would be the Pokémon found in ‘Pokémon Go”, in which the game 
places the pre-generated Pokémon into the live video image on your phone regardless of 
location or anything in the image. An example of the device finding a trigger image to 
manipulate can be found in any of the many “face changing” applications, such as ‘‘Face 
Changer” or “Funny Face Changer”, in which the trigger image is a human face, which the 
application then overlays with anything from a beard to animal ears. AR has reached hundreds of 
millions of users, thanks to its accessible nature on our everyday devices, as well as the 
simplicity in its user interface, so as not to alienate the inexperienced user (Kim, Kim, & Song, 
2019). 


Outside of the definition regarding digital overlay enhancing the real environment, AR 
has also been characterized by its accessibility and simplicity, needing in many cases, only a 
smartphone for hardware. Due to the accessibility and simplicity of AR, there have been many 
studies looking into how to incorporate the AR experience into the entire K-12 educational 
program through college (Garzon & Acevedo, 2019). In particular, the education of younger 
students in Early Childhood Education programs has been extensively looking into the use of AR 
to create immersive learning environments, create specialized learning programs for science, 
math, and reading, and even using AR for Behavioral Skills Training (Beck, 2019). The creation of 
an AR-aided learning program for special population groups with learning disabilities has also 
repeatedly been examined, as it cannot be overstated that the benefits of the simpler user 


interface can allow for a much wider variety of users than traditional learning programs (Barton, 
Pustejovsky, Maggin, & Reichow, 2017). While there are many more benefits of AR, and these 
will be examined in this review, the accessibility and minimal hardware needs are so engrained 
in what AR is, that leaving out these points when defining AR would be imprudent. 


Mixed Reality. Mixed Reality, or MR, refers to the merging of a virtual world with the real 
world, in which there is a physical component(s) in the real-world space whose 
actions/movements interact with an object(s) in the virtual world (Frank & Kapila, 2017), such as 
a physical joystick moving a virtual claw on screen. The use of MR has also been referred to as a 
Mixed Reality Learning Environment when referring to complex and dynamic systems that have 
been used for educational means (Chang, Lee, Wang, & Chen, 2010). Additionally, MR has been 
viewed as a combination of AR and what is considered full VR, with one part of the process 
taking place in the real world (AR) and the other part taking place in the virtual world (VR) 
(Weng, Rathinasabapathi, Weng, & Zagita, 2019). 


Defining MR can be difficult, as though the concept of MR is a bit scattered. Thinking of MR in the 
sense of a simulator can simplify the definition. One of the most widely understood simulators is 
the flight simulator, which has been examined in MR research from everything from the widely 
accessible ‘Microsoft Flight Simulator” (Korteling, Helsdingen, & Sluimer, 2017) to the simulator 
technology used by the U.S. military to train fighter pilots (Harper, 2015). 


While the scale and quality of the MR technology can vary greatly, the consistency of having a 
physical component interacting with virtual objects remains consistent. MR creates an immersive 
learning experience for individuals, and the use of incorporating physical components into the 
learning process has been found to create a greater impact over traditional technology training 
methods (Arango-Lopez, Ceréon Valdivieso, Collazos, Gutiérrez Vela, & Moreira, 2019). 


Virtual Simulation Environments versus Virtual Worlds 


Simulation technology has come a long way over the years, and while conversations 
regarding simulation technology now imply a virtual world, this is not always the case (Peterson, 
2010). Many simulations using VR for training purposes are scripted in nature, taking place ina 
relatively small virtual simulation environments compared to that of their virtual world 
counterparts (Kim et al., 2009). These virtual simulation environments can offer individuals the 
opportunity to train for a specific purpose in an environment that is more pleasant or inviting 
than that of the real-world environment (Burstin & Brown, 2010). In a virtual world, the size of the 
virtual space is much larger than that of a virtual environment, which facilitates the ability for the 
user to move through the virtual world in some capacity (Freitas & Neumann, 2009). While the 
virtual simulation environments are scripted, virtual worlds are open-ended. The virtual worlds 
allow for learning in both formal and informal approaches, as the user could learn through 
exploration of the world and through social interaction with those in the world (Freitas & 
Neumann, 2009). The size of the virtual world, and the ability to navigate it are imperative for 
training purposes that look to teach the user a skill related to their movement, as, for example, in 
a scenario that is looking to teach how to evacuate an area (Feng, Gonzalez, Amor, Lovreglio, & 
Cabrera-Guerrero, 2018). 


Virtual World Simulations 


Since the defining characteristic of virtual environments vs virtual worlds is size, there is 
still overlap and grey area. One example of this is a virtual car environment, if the user is placed 
into a virtual driver’s seat, they can effectively travel an entire virtual world while only being 
able to interact with the interface inside the car (El Saddik, Mahfujur Rahman, & Anwar Hossain, 
2008). There can also be scripted scenarios that take place over the span of large virtual worlds 
(Davis, Hercelinskyj, & Jackson, 2016), using virtual worlds as large as Second Life to achieve a 
scenario objective. These applications are referred to as Virtual World Simulations. Just as the 
example of learning how to evacuate an area can taught by allowing a user to navigate a virtual 
world (Feng et al., 2018), the evacuation process can also be taught in a virtual world through a 
guided simulation (Lochhead & Hedley, 2019). One form of virtual world simulations comes from 
the military, and is referred to as “Live, Virtual, and Constructive” (Strachan, 2016). 


Live, Virtual, and Constructive. Live, Virtual, and Constructive (LVC) training refers to three 
distinct types of simulation training, which when used collectively form the LVC model 
(Strachan, 2016). This LVC model is increasingly being implemented, particularly in military 
training. 


Live simulation training refers to the use of real equipment used by real individuals, such 
as guns, ships, or planes that are used against simulated non-enemy targets (Strachan, 2016). 
The use of live simulation training has been the traditional training model for the military for 
most of history, as the ability to accurately simulate warfare with technology has only been a 
recent advancement. Virtual simulation training refers to the use of a training device that can 
provide a replica of the equipment the individual would use in real combat (Antal, 2013). This 
equipment can be anything from a firearm that must be the same size and weight to allow for 
muscle memory training, to that of the inside of an armored vehicle that must be the same size 
and layout. 


Constructive simulation training refers to simulation training on a larger scale than an 
individual, where simulations operate without any direct one to one input of individual to avatar 
(Strachan, 2016). These constructive simulations are tactical in nature and allow for large number 
of men, vehicles, and equipment to be simulated in war games format to train for the movement 
of units through a virtual world (Antal, 2017). 


While these three forms of training are very different in their respective natures, they 
have been used in collaboration with each other to form a more effective training model (Mahon, 
2019). The use of live training simulations has been shown to be effective, however also costly, 
and not practical for all types of training (Best & Rice, 2018). When it comes to training pilots, the 
use of fuel and maintenance on the aircrafts can be costly, and firing at other aircraft (to train for 
surface to air combat scenarios for the pilots) is impractical (Best & Rice, 2018). The use of virtual 
training simulations are used to make up for the costs and limitations of live training simulations 
while still teaching the equipment usage, judgement, and decision-making skills that the military 
needs their soldiers to be trained on are still learned through the virtual training simulations 
(Mahon, 2019). 


The virtual training simulators have also allowed for the soldiers to not only train for 
general abilities and use of their equipment, but also allows for training for specific missions. 
Virtual training simulations allow soldiers to train in virtual environments created to replicate the 
real-world environment in which their upcoming mission will take place (Gervais, 2018). The 
ability to allow for planning and rehearsal of a mission in a simulation can mitigate potentially 
life-threatening mistakes, that can be avoided with virtual training simulations (Gervais, 2018). 


Finally, the constructive simulation training is not geared toward the individual soldier 
but can be used up the chain of command as far as an Army General (Strachan, 2016). The use of 
planning formations and large troop movements have been around for as long as war itself, 
historically viewed as large maps sprawled out on tables while figures representing units are 
moved around. The constructive simulation training of today allows for accurate movement 
depiction of any number of military units, in any condition, on any scale (Strachan, 2016). The 
combination of these simulation training types have formed the LVC method that has taken root 
in the military today. Using the different types when appropriate, the military can get the most 
out of its resources while not reducing the quality of training that its soldiers receive (Mahon, 
2019). 


Equipment 


Screens and Displays 


To have a virtual reality experience, two things are present to the user: the sight of and 
sounds of the virtual reality (Howard, 2019). For portraying the visuals of the experience, three 
general forms of equipment are utilized: a screen, a headset, or a CAVE system (Ritz & Buss, 
2016). Ascreen, being the simplest and most widely used, can come in the form of a phone 
screen being used to project an AR overlay onto the world (Barton et al., 2017) or to a computer 
screen displaying a Massively Multiplayer Online Role Playing Game or “MMORPG” such as 
Second Life (Kim et al., 2009). One advantage that the traditional screen set up has over others is 
the use of extra sensory data, such as being able to use eye tracking equipment in conjunction 
with the virtual reality software (Aguileta, Brena, Mayora, Molino-Minero-Re, & Trejo, 2019). As 
something like the eye tracking equipment is required, to have the user facing in the general 
direction of the screen with their eyes visible, the use of a headset that would block the eyes 
from the tracker or a set up in which the user would be moving their head away from the eye 
tracker would both impede the eye tracking equipment’s ability to function (Aguileta et al., 
2019). 


The second option of display, virtual reality headsets, are sometimes referred to as Head- 
Mounted Displays or “HMDs” (Alsaeedi & Wloka, 2019). These headsets are essentially goggles, 
that, due to their weight, are attached to the user’s head via a tightening strap that goes around 
the sides and back of the user’s head (Martelli, Xia, Prado, & Agrawal, 2019). Due to some users 
experiencing a lack of comfort while using the headsets, a smaller more lightweight option of 
“VR glasses” have seen a rise in development (Yu, Zhou, Wang, & Zhao, 2019). For the display 
the headsets either have a singular screen that is placed in front of the user’s eyes, or 
alternatively two smaller screens with one in front of each eye, the VR glasses always have the 
two screens for each eye (Yu et al., 2019). While extra equipment, such as the eye tracking 


equipment mentioned previously, was not immediately available for use in a VR headset setting, 
there have been advancements towards bringing all these technologies together. Recently real- 
time eyeblink detectors have been researched using VR headsets, which as the technology 
increases in use these additional data points will become more prevalent (Alsaeedi & Wloka, 
2019). 


Lastly, going from the most accessible to the most specialized form of getting the visuals 
of VR to an individual, there is the Cave Automatic Virtual Environment or “CAVE” system 
(Merchant, Goetz, Cifuentes, Keeney-Kennicutt, & Davis, 2014). The CAVE displays the VR 
environment to the user by projecting the visuals onto the walls of a room, this can be anywhere 
from three out of four sides of a square room, to all sides of a room, to even the four sides along 
with floor and ceiling for a completely enclosed virtual space (Cayley & Lemmerman, 2006). The 
CAVE, while costly, does provide distinct differences over the use of seeing the virtual 
environment through a headset device. The most notable difference is the difference that comes 
from spatial reasoning on the individual using the device from the impact of being able to see 
their own body (Lassagne, Kemeny, Posslt, & Merienne, 2019). While using a headset display, 
the user either is wearing gloves or holding a device so that the headset can show the user 
where their hands are in the virtual space, however with the CAVE, none of this is necessary 
which should be an advantage (Lassagne et al., 2019). However, the headsets have had the 
advantage of time and more wide use for fine tuning. CAVE users often report seeing objects as 
too close or too far when they reach for them with their hands (Lassagne et al., 2019). While not 
an immediately perfect system, the CAVE offers the user the ability to walk through the virtual 
space freely on their own without extra equipment to simulate the walking experience. 


Walking in Virtual Reality 


While the CAVE system offers a solution to walking in a virtual space without walking 
equipment, this equipment is available for screen and headset virtual displays (Jungwon et al., 
2010). While walking is most natural movement for an individual, it does pose a challenge for VR 
in which the space allowed for the individual can be as small as a single room (Jungwon et al., 
2010). This obstacle has been overcome, in part, through the use of Locomotion Interface or “LI”, 
which consists of four different types: planar treadmills, passive user walking devices with a 
turntable, sliding devices with mobile robots, and programmable foot platforms with rotational 
capability (Jungwon et al., 2010). For the purposes of this review, each type of treadmill and 
motion platform will not be individually covered, as advancements in LI have been as almost 
exponential in development, going from three degrees of freedom in 2010 Jungwon et al., 
2010), to six degrees of freedom with motion platforms in 2015 (Sinitski, Lemaire, & Baddour, 
2015), to the newest form of motion platform, the omnidirectional Platform that can theoretically 
offer 360 degrees of freedom (Monroy, Lutz, Chalasani, & Smolic, 2018). While this review will 
not go into the specifics of the design of these LI, it is important that the advancements in virtual 
reality walking technology be addressed. 


How Virtual Reality and Simulations Can Improve Leaming 


The use of VR and simulations have been utilized in a variety of multidisciplinary 
scenarios for different learning environments, from distance learning to therapy treatments 


(Correia et al., 2014). VR and simulations have also been used in a wide variety of purely 
educational environments. VR and simulations have been found to aid in the learning of a second 
language in children (Schwienhorst, 2002). Additionally, these systems are effective in learning 
outcomes of children throughout the entire K-12 system, as well as higher educational settings 
(Merchant et al., 2014). VR systems were found to increase student involvement and self-efficacy 
in some studies (Georgiou & Kyza, 2018). Of note, VR systems have been an effective educational 
tool for those that have a physical handicap, such as cerebral palsy (Kirshner, Weiss, & Tirosh, 
2011). 


While research has been done on the benefits and applications for a traditional learning 
environment, there experimental learning techniques involving VR and simulations that are still 
being discovered. These include social benefits from the use of role-playing simulations in a 3D 
virtual environment to facilitate interviewing and diagnostic skills of a counselor (Lowell & 
Alshammari, 2019). 


Creation of Environments and the Elimination of Distance 


One of the biggest draws for VR and simulations in the learning environment, is the 
ability to create a specific virtual space for the individual to learn in that would otherwise be 
impractical (Correia et al., 2014). While VR simulations can seem impractical due to cost, their 
reuse makes them more cost efficient when compared to live simulations such as battle 
simulation, which can make the large amount of equipment needed. When reusability is 
considered, VR simulations could reduce the cost needed for training (Fuentes, 2018). The 
ability to create specific locations for training in the military can also allow for the creation ofa 
war zone in preparation for an upcoming mission (Joy, Rykard, & Green, 2014). This benefits the 
soldiers as the area is no longer a completely unfamiliar territory even when they are arriving 
for the first time (Colameo, 2016). When the VR environments are connected over a network, the 
elimination of distance issues for learning from or training with other individuals across the 
globe is a reduced (Umoren et al., 2017). This is empowering it itself as the ability to connect 
individuals for learning or any team-based training has a multitude of benefits on its own. 


The virtual environments created can also get around the issue of needing a large open 
space for training, such as being able to train emergency personnel in the aftermath of a natural 
disaster, which is much easier to replicate using VR and simulations rather than the expensive 
cost of making a replica of an area hit by such a disaster (Fung et al., 2015). These disaster 
replications have been utilized for not only natural occurring disasters, but also for simulating 
the emergency preparedness, response, and mitigation of non-natural disasters such as a 
nuclear event (Davis, Proctor, & Shageer, 2016). 


The creation of virtual environments further takes out the danger of training individuals in 
these environments that would be hazardous in a live training environment, even as far as being 
able to train marines in combat exercises with no risk to themselves as all the potential danger to 
the trainee is mitigated by the nature of VR (Fuentes, 2018). Learning in a specific environment 
has also been found to be beneficial to students (Georgiou & Kyza, 2018). 


This locational learning can be costly and inconvenient in the real world to travel toa 
specific location for learning purposes (Moorhouse, tom Dieck, & Jung, 2019). However, using 
VR and simulation, students can travel to see wonders, historical place, and museums to learn 
without ever leaving their classroom, while benefiting from locational learning (Moorhouse et 
al., 2019). 


Team Training 


The use of team training with VR and simulations is an important issue and comes with its 
own list of benefits that are exclusively from the impact of team training (Punnarumol, 2015). 
Improvement of team performance (Eppich, Nannicelli, Seivert, Sohn, Rozenfeld, Woods, & Holl, 
2015), reduction in time for teams to plan and begin the task required (Murphy, Curtis, Lam, 
Palmer, Hsu, & McCloughen, 2018), improved communication (Zemliansky, 2012), improvement 
in team leadership skills (Rosenman, Vrablik, Brolliar, Chipman, & Fernandez, 2019), and 
improved team member satisfaction (Han, Chae, Macko, Park, & Beyerlein, 2017). See “Team 
Training” section in this document. 


Use of Al with Virtual Humans 


With the notion of benefits of team-based training, there is also the use of VR and 
simulations to replace teammates with AI, which have also been referred to as “synthetic 
teammates,” “agents,” “pedagogical agents,” or “virtual humans.” The use of virtual humans has 
been studied with success in many different learning environments, such as the K-12 system 
(Schroeder, Adesope, & Gilbert, 2013). 
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The virtual humans used vary greatly in their properties such as in appearance, gestures, 
movements, and speech (Craig, Gholson, Driscoll, 2002; Craig & Schroeder, 2018). More 
animated and lifelike virtual humans have found to improve learning over those that are more 
static in nature (Craig, Twyford, Irigoyen, & Zipp, 2015), however even just the virtual human’s 
ability to converse has been studied in depth. Learning software such as AutoTutor has been 
using virtual humans to hold natural conversations with students to facilitate learning (Graesser 
et al., 2004). This is done through virtual on-screen characters, that, through conversation, can 
direct the flow of instruction and facilitate learning (Schroeder et al., 2013). 


Virtual humans are not restricted to the learner/teacher dynamic and have also been 
utilized as a virtual replacement for a member ofa team. One such example being the 
development of interpersonal coordination using a virtual rowing teammate (Varlet et al., 2013). 
The virtual reality rowing experience was equipped with virtual teammates that the user had to 
synchronize their efforts with, which was found to transfer to real teammates later, showing 
promise for VR training in many physically coordinated team efforts (Varlet et al., 2013). Virtual 
teammates have also been found to help the improvement of skills in the medical field (Djukic et 
al., 2015). Utilizing virtual teammates, live medical students, and virtual nurses (and vice versa) 
were paired in a virtual environment and shown to develop skills no faster than live student and 
live nurse pairs (Djukic et al., 2015). These studies show that if a real human in unavailable for 
learning or training purposes, that the use of a virtual human in VR and simulation environments 
can provide an adequate substitute. 


Improvement on Skills and Abilities 


While VR and simulation technology can be impressive, the impact that technology has 
on the individual’s ability to learn from the experience is the most critical question for educators. 
Studies have shown a variety of beneficial learning outcomes for students that have utilized the 
new technology, from both traditional learning to teaching applied skills. In educational 
environments, students have shown improvement in primary mathematics education while using 
mobile AR instruction materials (Chen, 2019). Improvements were also shown in other STEM 
related lessons, such as in successful collaboration when given tasks in a virtual learning 
environment designed for teaching about electronics (Zhen, Xing, & Zhu, 2019). 


The ability to bring high-school students to a location virtually has also shown a positive 
immersion impact on motivation and conceptual learning (Georgiou & Kyza, 2018). Outside of 
education, VR and simulation technology has also shown improvement in industries looking to 
train applied skills. These skills can be as advanced as training to optimize military physicians in 
surgical care units (Ka-Chun et al., 2016), to more traditionally labor focused skills in the 
construction industry (Goulding, Nadim, Petridis, & Alshawi, 2012). 


Reduced Cost 


The reduction of the cost of training is an immediately quantifiable benefit of VR and 
simulation technology. The more expensive that the cost of training an individual is in a real-life 
environment, the more of an investment the VR and simulation technology can be. This savings is 
easily recognized in the costs of high-end military equipment (Best & Rice, 2018). The high cost 
of ammunition, fuel, and maintenance to equipment is not a factor when the training is taking 
place in a virtual environment. Having to procure specialized equipment for training can also be 
problematic. 


Medical training, for example, has seen a large cost reduction in training from the 
replacement of medical cadavers with the implementation of VR and simulation training (Allen et 
al. 2016). Medical cadavers are not only expensive and logistically troublesome to transport and 
preserve but are also sold in a limited supply due to their nature. The use of training simulators 
for practicing surgical techniques can eliminate this expense and supply issue (Allen et al., 
2016). Additionally, the space available for training can become an expense as well, such as 
when large open areas are required for vehicle training (El Saddik et al, 2008). Particularly ina 
city environment space can be limited and expensive, the VR and simulation technology can 
make a very limited space become a practical training environment. 


Cost reduction has not only been seen in the applied industries, but also in traditional 
education environments. STEM in particular has been given a large amount of attention in the VR 
and simulation environment, as there has been reported disappointment in students attending 
STEM classrooms expecting the learning environment to be filled with the best equipment and 
technology (O’Leary, Shattuck, & Kubby, 2012). This disconnect of expectations from the reality 
of what a school can provide, can cause students to become disengaged from learning. VR and 
simulations address this issue by allowing students to connect to remote laboratories online, 


allowing them to engage with virtual equipment that a school would never be able to afford 
otherwise (Garcia-Zubia et al., 2017). 


Convenient and Reproducible Training 


In a similar vein to reducing the cost of training, VR and simulations can make the training 
more convenient. Aviation has noted the use of VR and even less immersive computers have 
made the training of pilots a more convenient process (Aoki, Oman, Buckland, & Natapoff, 2008). 
This convenience is even more exemplified outside of the military, such as being able to train 
pilots and air control personnel without the need to tie up a terminal that would otherwise be in 
use at a civilian airport (Littlepage et al., 2016). 


Medical training has also seen the benefit of convenience through not only the 
elimination of the logistics of medical cadavers, but also the quality fluctuation of said bodies 
(Allen et al. 2016). Medical cadavers can be the bodies of deceased elderly whose bodies have 
changed with age, or people who have died due to an illness or condition that has affected their 
body in such a way that it had resulted in death. As such, medical cadavers do not offer a full 
representation of the bodies that surgeons would be working on, in particular surgeons in the 
army who would for the most part be working on relatively healthy individuals that are within 
military fighting age (Allen et al., 2016). The use of VR and simulations can offer a better 
representation of what the surgeons target demographic for bodies would be, rather than the 
more limited real supply that is available in medical cadavers. Additionally, the quality of the 
virtual body would be consistent across the training for any number of surgeons, as its virtual 
nature makes it endlessly replaceable, guaranteeing that all have the same training experience 
available to them. 


Specialized Populations 


There are many individuals that, for a variety of reasons, struggle with traditional 
learning. While the review has discussed VR creating safe learning environments that would 
normally be hazardous for an individual, VR can also create safe environments that would be 
safe for most individuals but are still dangerous for those with particular conditions (Yamaguchi, 
Foloppe, Richard, Richard, & Allain, 2012). Of particular note, the Alzheimer’s community has 
been examining the use of VR-based training to enhance the autonomy of Alzheimer’s patients 
when it comes to being able to cook for themselves (Foloppe, Richard, Yamaguchi, Etcharry- 
Bouyx, & Allain, 2018). This has also been studied in the case of adults with dementia (Hill et al., 
2017), facing similar cognitive challenges to those with Alzheimer’s, such as being able to live 
independently so that there can be an increase the quality of life metrics, as the need to be able 
to maintain and retrain common tasks is critical for these individuals (Foloppe et al., 2018). 


For others, their reasons can be physical in nature, such as those that suffer from a 
condition like cerebral palsy. For them, VR, and simulation technology offer an adaptive method 
of learning (Kirshner et al., 2011). For others, such as children diagnosed with special needs, VR 
has been used as an additional learning tool for increasing the level of interaction that these 
children have in their education (Cai, Chiew, Nay, Indhumathi, & Huang, 2017). In more 
manageable cases, students with ADHD have also been shown to benefit from the 


implementation of computerized testing and virtual classrooms over that of the traditional 
learning environments (Parsons, Duffield, & Asbee, 2018) Unfortunately, the long-term 
effectiveness in the use of VR for teaching these specialized populations has not been fully 
researched (Chia & Li, 2012), and it is noteworthy that the long-term effectiveness of VR has not 
been studied outside of these populations. 


Limitations 


A general limitation of the training research is that it only investigates the immediate 
effects VR or simulated training and not sustained or long-term benefits. Some of the research 
has examined the reduction in the outcome of VR training over time, which shows a significant 
decrease over the course of three months (Van De Ven et al., 2017). Even these studies that show 
the decrease in performance after the initial training do not compare the reduction of 
performance from a live training session, making the impact of VR over live training unknown 
over time (Van De Ven et al., 2017). 


The skills learned by the individuals in the virtual world have not been fully researched 
to see if they transfer over to real world applications. While there have been studies that attempt 
to show VR and simulation training improve expertise and provide a more effective transfer of 
learning to other environments. Some findings have only been measured in the improvements 
that the user gains while in the VR environment (Rezazadeh, Wang, Firoozabadi, & Hashemi, 
2011). While there are some instances where the training in the virtual space doesn’t require a 
learning transfer, such as studies that have looked into the ability to train individuals for 
leadership roles in virtual environments for the purpose of working with a virtual team (Brewer, 
Mitchell, Sanders, Wallace, & Wood, 2015), the overall goal of training in a virtual environment is 
to be able to transfer that skill and knowledge over to the real world. While implied that skills 
developed in a virtual environment would transfer, there is at best an imbalance of evidence to 
support this compared to the popularity that training in VR simulations has acquired (Goode, 
Salmon, & Lenné, 2013). 


While the VR and simulation technology can reduce cost in some ways, the “top of the 
line” pieces of equipment can be expensive. While this review has gone over many studies that 
have discussed the value of this technology, there have also been studies that have shown 
inexpensive methods are a suitable alternative for training purposes (de Siqueira et al., 2017). 
Such is the case for the practice of laparoscopic surgery, in which surgical trainees were found to 
have no improvement or advantage when given the use of virtual reality simulators over those 
that were given ‘“‘take-home” box trainers (de Siqueira et al., 2017). While both the virtual reality 
simulators and the box trainers are cheaper and more convenient than medical cadavers, the 
virtual reality simulator can be hard to justify in its expense compared to a training device that 
can be composed mostly of cardboard. 


Application of Virtual Reality and Simulations 


While the limitations of VR and simulation technology are present, there have been many 
areas that have implemented the technology with some success. Areas that have had large 
impacts or incorporation of VR and simulation technology is discussed below. 


Mhittary 


Almost every level of the military has incorporated some level of VR and simulation 
technology into it training (Strachan, 2016). The military founded the LVC training model 
mentioned in this review, which is geared towards using VR and simulation technology for 
training soldiers with virtual simulation and training the highest generals directing large scale 
operations (Antal, 2012). The Marine Corps have begun the testing of simulated networks for 
training Marine Air-Ground Task Forces as of 2018, employing a collaborative system level of 
training (Fuentes, 2018). In addition to ground soldiers, the mechanized portions of the military 
have also utilized the LVC training model to train Navy, (Strachan, 2016), Airforce (Mahon, 2019), 
Tank divisions (Mahon, 2019), and Field Artillery (McKiernan, 2013). The expense saved by 
having these training exercises take place in a virtual environment is substantial and removing 
the potential for hazardous training accidents makes the military a prime target for the benefits 
of VR and simulation technology (Colameo, 2016). 


Medical 


While the impact of many VR and simulation uses have a mitigation of hazards to the user, 
the medical fields benefits are different in that the hazard mitigated is for the patient (Consorti, 
Mancuso, Nocioni, & Piccolo, 2012). The use of these “virtual patients” have been introduced in 
training as an alternative to traditional means of practice, removing actual patients from the 
possibility of harm while the medical practitioner is still in their educational process (Consorti et 
al., 2012). The use of VR simulators for specialized training, such as endoscopy programs, have 
seen an increasing amount of integration with simulation-based training (Khan et al., 2019). The 
simulators allow for medical practitioners to practice their surgical techniques, with no risk of 
patient harm or discomfort (Khan et al., 2019). Additionally, nurses have been trained with VR in 
long-term patient care practices (Gdanetz et al., 2018). The ability for nursing staff to train online 
using virtual patients also allows for more potential students to enroll in training opportunities. 
As there have been repeated calls for more highly trained nursing staff, increasing the number 
of potential nursing staff will help the already strained medical system (Gdanetz et al., 2018). 


The use of VR and simulation technology can even be used by patients themselves. In the 
area of therapy and rehabilitation, patients that have had difficulties with coping in the real world 
are given their problems gradually in a virtual and very controlled environment (Burstin & 
Brown, 2010). The ability to manipulate the visual, audible, tactile, and any other part of the 
interactive experience, gives the therapist and patient control over the scenario that would 
otherwise be difficult to create (Burstin & Brown, 2010). By creating a more inviting environment, 
the therapists have an additional set of tools to aid them and their patients through recovery. 


Crisis Intervention Teams 


Crisis Intervention Teams (CIT), are police and mental health collaboration teams 
specializing in involvement of those with mental illness (Crisanti, Earheart, Rosenbaum, Tinney, 
& Duhigg, 2019). Research has shown up to a fourth of individuals suffering from a mental health 
problem also have a history of police arrest, and the additional time needed to interact with 
these individuals in a caring way can take up resources and officers’ time (Crisanti et al. 2019). 
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Through the use of simulated training exercises, officers have been able to train and adapt their 
techniques in an environment that is not only safe for them, but they are also able to train without 
the ethical implications regarding involving those suffering from a mental health issue 
(Stanojevic & Stanojevic, 2016). The training simulations have been created through a joint effort 
of psychiatrists, crisis intervention unit detectives, and crisis specialists (Crisanti et al, 2019). 
Using this specialized virtual simulation training, officers have become more proficient when it 
comes to their involvement in this very vulnerable population (Stanojevic & Stanjevic, 2016). 


Nuclear Disaster Response Teams 


The impact of a nuclear disaster can have severe consequences for a large portion of the 
population, as have been seen in the aftermath of Fukushima and Chernobyl (Davis, Proctor, & 
Shageer, 2016). The large scale of the potential disaster zone requires the use of conceptual 
models for planning and assessing the safety needs of those in the affected areas (Davis, Proctor, 
& Shageer, 2017). The use of LVC simulation technology, in particular, the Constructive, has 
been utilized for creating, simulating, and then evaluating the many possible hurdles that a wide 
range of emergency response teams will need to overcome in this potential time of crisis (Davis 
et al., 2017). Virtual training has also been designed for existing nuclear reactors, with the focus 
on mitigating a potential meltdown scenario (Davis et al., 2016). 


Mining and Drilling 


The mining and drilling industry leaders have also been exploring the use of VR and 
simulation training for some time (Neustupa, Danel, & Repka, 2011). While the traditional image 
of a miner with a pickaxe might be the first image to come to mind when thinking about coal, the 
state of coal mining has become much more technical in nature (Neustupa et al., 2011). In 
opencast mines, the process is almost all technology-based from beginning to end. From the 
excavator for mining, to the distance belt for transportation, the process of operating a mine is 
becoming more centralized at a control station environment (Neustupa et al., 2011). Mines such 
as these have been utilizing VR operation systems rather than training traditional machine 
operators, which in turn means that VR and simulation training for the skills needed of those 
working the control stations has been utilized by the coal industry. 


However, not all these operations have become automated, as many still rely on the 
physical presence of people and the labor they provide. In those environments, VR and 
simulation technology has been used not only in the training for using equipment, but to train for 
the safety of those operating (Mehdi Naqvi, Raza, Ybarra, Salehi, & Teodoriu, 2019). Through 
simulation-based training, the goal of decreasing human error in environments such as offshore 
drilling can have an immense impact on the reduction of serious injuries or even fatalities (Xie, 
Yang, Wang, & Wang, 2018). 


In addition to training for the day to day operations, emergency simulation training has 
also been introduced to safely train the drillers for the potential accidents that can take place 
(Musharraf, Khan, & Veitch, 2019). 


Construction 


With working environments containing heights, power tools, and heavy materials, 
construction safety has naturally been a concern (Norris, Spicer, & Byrd, 2019). Traditional 
means of safety training have also been met with dissatisfaction from construction workers, who 
have had serious concerns about the effectiveness of the training (Norris et al., 2019). The use of 
virtual reality allows construction workers to reinforce the safety training they have received 
with practice in a safe virtual environment, such as the danger of falling while on a plank or 
beam (Shi, Du, Ahn, & Ragan, 2019). This has been done in a variety of ways, such as the use ofa 
motion tracker to give feedback to the worker, observing an avatar demonstrating correct 
movements, and the ability to practice their movement in the virtual space (Shi et al., 2019). 


The construction of materials offsite, while initially less dangerous, has also been looking 
to VR for a safer way of training their workers (Goulding, Nadim, Petridis, & Alshawi, 2012). VR 
and simulator training have also made their way into the design and development stages of 
construction (Hill, 2016). Like the constructive portion of LVC, the virtual simulators allow for 
construction designers to plan out the building, easily modeling and manipulating the structure 
in a virtual space (Hill, 2016). 


Customer Service 


While the above industries have a hazard involved that the VR and simulation technology 
can mitigate, there are industries with no such hazards that have also sought this technology. 
One such example is the industry of product development and customer service, which have 
looked to virtual worlds to offer new methods of gathering information (Pridmore & Overocker, 
2014). For customer service, some of the industry is looking to replace the process of service 
level requests of provider-to-customer to a completely digital computer-to-customer interface 
(Bi et al., 2017). These computer-to-customer interactions take the form of virtual agents, able to 
interact with customers on an autonomous level (Kerr & Bornfreund, 2005). However, the 
practices, while successful on a technical level, have been viewed by customers as an 
exploitation of trust by some (Kerr & Bornfreund, 2005), and a risk to privacy by others 
(Pridmore & Overocker, 2014). 


Future of Virtual Reality and Simulations 


The potential for VR and simulation technology seems like and endless well of 
possibilities, and the impact of virtual environments has been a heated discussion topic in the 
learning community (Doumanis, Econmou, Sim, & Porter, 2019). The ability to connect to others 
in virtual communities of learning has been a sought-after goal for many organizations 
(Akoumianakis & Alexandraki, 2012). Everything from knowledge, operational skills, or the 
ability to lead others has been examined through the eyes of VR and simulation (Levesque, 
2012). 


Despite the growing desire to embrace technology and all its wonders with open arms, 
there must be cautious advancement using critical analysis. While the technology is enticing, not 
all training has been found to be more successful in the virtual worlds (Negut, Matu, Sava, & 


David, 2016). Despite this, the trend of technology is moving forward. As personalized and 
wearable devices become more powerful, more advancements in technology will be used 
before they are empirically tested (Xie, Chu, Hwang, & Wang, 2019). With the tide of VR and 
simulation technology increasing exponentially, the only thing to do is to have rigorous 
empirical evaluations, to weed out the ineffective technology from the technology that will 
advance the world. 


VR and Simulations in MO0Cs 


Given the nature of MOOCs trying to reach out to a wide audience, the use of high-end VR or 
AR equipment is not a practical application. Instead, many MOOCs focus on computer-based 
simulations to offer a more interactive learning experience (Song et al., 2019). The most 
common applications for which are in the STEM fields, with research showing the use of 
simulation in MOOCs for chemistry (O’Malley, Agger, & Anderson, 2015) and quantum 
mechanics (Freericks, Cutler, Kruse, & Vieira, 2019). The use of simulation has also been utilized 
to test the MOOC networks, examining not only the learning elements but also the social 
constructs able to create in the MOOC ecosystem (Zhang, Skryabin, & Song, 2016). However, 
the empirical research regarding MOOCs and simulation use is still lacking, and the research 
regarding MOOCs and VR/AR is even more scarce. Further research in this area is needed. 


Blended Learning 

Blended learning (BL) environments are those that combine face-to-face learning with online 
learning. There is no clear-cut standard for the mixture of face-to-face and online opportunities 
that qualifies a course as being in the blended learning format (Graham & Dziuban, 2008; 
Millichap & Vogt, 2012; Stacey & Gerbic, 2008). BL environments differ widely in several 
important features, such as technology usage, the amount of online activities available to 
students, and the degree to which the online portion of the program is intended to replace 
classroom activities (Smith & Kurthen, 2007). In general, BL has several consistent attributes, 
some learning opportunities are available online, some learning happens in the traditional 
classroom, and the online and traditional learning are complementary to one another (Panopto, 
2019). In the Horizon Report, Becker et al. (2017) called blended learning (along with mobile and 
online learning) a ‘foregone conclusion’ and that its use at colleges and universities is on the 
rise. Blended learning offers flexibility, ease of access, and the use of technology to enable 
learning. Becker et al. (2017) state that findings from blended learning show students 
experience an increase in creative thinking, tailored learning, and independent learning. 


Smith and Kurthen (2007) suggested that an appropriate taxonomy for classifying online learning 
discriminates four distinct levels: 1) web-enhanced, which encompasses courses that use 
minimal web-elements, such as syllabus or announcement features on a learning management 
system, 2) blended, which has additional online documents but hosts less than 45% of course 
activities online, 3) hybrid, which offers 45% to 80% of class activities online, and 4) fully online, 
which offers greater than 80% of activities and content online. Trentin and Bucconi (2014) 
suggested that the emphasis in blended or hybrid learning shift from a space and time 
dimension to a focus on integrating different teaching methods and tool usage, preferring the 
term ‘hybrid solutions’. In their view, viewed hybrid solutions in three dimensions. These are 


the dimension of the 1) learning process, which may be experienced either individually or 
collaboratively, 2) the setting of the learning events in either the classroom or outside of the 
classroom, and 3) the learning space, which is either onsite or online. The combination of the 
three elements provides a unique learning path that can utilize online interaction, depending 
upon the course pedagogy. 


For example, onsite activities and goals should lay the groundwork for online activities by 
providing assignment and goal clarification, elaborating on expected results or deadlines, and 
exposing questions or prior knowledge deficiencies. As illustrated below, the combination of 
onsite and online environments in the realm of individual or collaborative interactions creates a 
2 x 2 matrix for considering possible relationships. 


In each of the quadrants, mobile or network technologies can play a significant role in the 
learning process. In quadrant one (represented by onsite/individual learning), technology can 
improve the communication process between instructors and students which provides improved 
opportunities to engage in knowledge exchange using tools like social media for formative e- 
assessments (Luckin et al., 2012; Trentin & Bucconi, 2014). Data collected using technology tools 
can be evaluated using learning analytics to improve assessment (Luckin et al., 2012). 


The second quadrant (online/individual learning) utilize technology to provide the requisite 
‘meeting place’ where learning occurs. Technology allows students to perform experiments 
using remote online labs and helps teachers track each student’s activities through data 
collection (Trentin & Bucconi, 2014). 


Quadrant 3 (online/collaborative learning) provides community interaction in social spaces and 
learning management systems using technology that brings students together with an instructor 
for collaborative activities in synchronous or asynchronous modes. These technologies increase 
students’ abilities to self-assess and self-help through interacting with group members in 
application problems that allow for sharing problem solving strategies and possible solutions. 
Instructors can assess students individually and in groups throughout the collaboration process 
through to the delivery of the product, and, in the process, observe individual learning outcomes 
(Trentin & Bucconi, 2014). Again, data collection and analytics become valuable, when 
combined with more subjective data (peer evaluation, teacher evaluation) to draw conclusions 
about both individual and group contributions (Trentin, 2009). 


Quadrant 4 (onsite/collaborative learning) uses technology to organize and manage in-class 
collaborations by allowing teachers to collect students’ group discussions and offer immediate 
feedback. This can be accomplished through group reflection activities on problem solving or 
concept learning in which the instructor can collect real-time data to both engage students and 
offer immediate feedback for student self-evaluation (Trentin & Bucconi, 2014). 


Learning Learning Space 
Process Onsite Online 


Individual Personal study at home, in the Personal study in virtual spaces 
library, or in the classroom like remote labs, interactive 
simulations, or immersive 
environments 


Collaborative Community learning at the library, Community interaction is social 
at home, or in the classroom spaces like social media, or 
collaborative virtual 
environments 


Based on Trentin & Bucconi, 2014 


The Success of Blended Learning 

The success of blended learning has ushered in the need for institutional policies and plans for 
guiding the implementation of the blended learning environment (Becker et al., 2017). These 
policies include plans for faculty development, strategies to make the necessary curricular 
changes, and financial appropriations for the switch to the blended mode of delivery (Becker et 
al., 2017). 


Teacher support should include models of best practices in blended learning and examples of 
course designs to aid instructors to re-design content for the blended mode of delivery (McGee 
& Reis, 2012). Teachers can tend to be suspicious of directives issued from administration and 
blended learning initiatives can cause stress to instructors who fear that the quality of the course 
may decrease or that they will lose intellectual property rights in the transition (Moskal, Dziuban, 
& Hartman, 2012). Dziuban, Hartman, Cavanagh, and Moskal (2011) found that one successful 
strategy for faculty training was to offer a professional development course through the blended 
format for 8-weeks (over 80 contact hours). In this course, the faculty members become students 
and can experience the blended context for themselves. The strategic support for faculty has 
been shown to improve faculty satisfaction with teaching blended course sections (Dziuban, 
Hartman, Cavanagh, & Moskal, 2011). 


The strategies of the blended learning environment must be integrated throughout the academic 
system to include the registrar, the teaching and learning center, and the technology centers for 
academic and IT concerns (McGee & Reis, 2012). Moskal et al. (2012) stressed that institutions 
tackling blended learning must have a robust infrastructure that can handle continuous change. 
Moskal et al. (2012) suggested that institutions endorse a planning strategy that addresses the 
following questions: 


1) Why should our institution engage in blended learning and what are our outcome 
expectations in both the short- and long-term? 

2) What student benefits are at the core of the incorporation of blended learning? Are we 
seeking to improve retention, learning outcomes, student success, etc. 

3) How will we choose the courses that are offered in a blended format? 

4) How will we influence faculty success? 

5) How will blended learning be implemented institution-wide? 

6) What time and financial investments are we prepared to make? 


Dziuban and colleagues noted that student support is of paramount importance in the move to 
offer blended courses. In this vein, it is critical to consider the aspects of student success, 
withdrawal, and perceptions (Dziuban, Graham, Norberg, & Sicilia, 2018). For students to 
succeed in blended learning environments they must possess the self-motivation necessary to 
succeed in an online environment. Student success in a blended learning environment is hard to 
predict; however, no relationship between ACT or SAT scores were predictive of a student’s 
ability to thrive in a blended learning course. However, current GPA was predictive. 
Demographically, females are more successful than males (88% vs. 81%) and success rates 
declined with age (Dziuban et al., 2011). 


Blended learning has been investigated through several meta-analyses or systematic reviews in 
different subject areas. Blended learning was found to have a consistent positive effect on 
learning compared to no intervention and was found to be least or more effective than 
nonblended instruction for acquiring health professions knowledge (Liu et al., 2016). Similarly, 
Li, Jing, Yuan, Chen, and Sun (2019) found that blended learning effectively improved nursing 
students’ knowledge level and satisfaction and stated that this learning mode could be 
successfully used in nurse training. Alammary (2019) found that blended learning has potential 
for enhancing novice programming students’ performance. Kozikoglu (2019) found that flipped 
learning had a positive effect on student motivation, academic achievement, meta-cognitive 
awareness, self-efficacy, critical thinking, attitude, information literacy, and retention. Ko (2019) 
also found positive benefits of flipped classrooms such as personalization through resources and 
teacher access, higher order thinking improvement through problem solving, and student 
collaborative learning through peer groups. Studies in which student collaboration and self- 


directed learning with remediation and application were employed showed the highest positive 
trends (Ko, 2019). Means, Toyama, Murphy, and Baki (2015) found that the difference in student 
performance was significantly better in blended learning environments over face-to-face 
instruction. Highlighted that blended learning environments studied in their meta-analysis 
revealed that blended learning modes were characterized by increased learning resources, 
increased learning time, and design elements that promoted interaction among the students 
(Means, Toyama, Murphy, & Baki, 2015). 


Dziuban, Graham, Moskal, Norberg, and Silicia (2018) expressed concerns about the 
conclusions of meta-analyses because the effect sizes are derived from a linear hypothesis 
testing model which assumes no correlation between treatment and error terms. Such 
assumptions can confound the meta-analyses by negating any variables that may be factors in 
the blending. Dziuban and colleagues (2018) also expressed concern that blends in the meta- 
analyses are not equivalent from study to study. For example, Dziuban and colleagues (2018) 
take exception to Means et al.’s (2010) inclusion of online instruction, email, class web sites, 
learning management systems, and laboratory assessments as representing blending. Having 
multiple blending configurations make effect statements suspect in meta-analyses (Dziuban, 
Graham, Moskal, Norberg, & Silicia, 2018). 


Morris (2010) expressed concern over the inclusion of varied instructional method, time, and 
curricular method used in some studies which could confound results on the effectiveness of 
blended learning. Morris (2010) determined that there was no statistically significant difference 
in performance between blended course participants and traditional face-to face environments 
when time, methodology, and curricular method are controlled (Morris, 2010). 


Dziuban et al. (2018) stated that the effectiveness of blended learning is centered around access, 
student perceptions of their learning environment, and success. Students were found to operate 
under a strong if-then decision-making schema to evaluate their educational experiences, and 
this schema was independent of course delivery method, the perceived relevance of the content, 
and the expected grade. Instead of using effect size to stipulate the value of blended learning, 
suggesting that student success, withdrawal rates, and perception of learning be the standards 
by which blended learning is measured. In their meta-analysis, found that blending improved 
success rates for most students, whether minority or non-minority. Students ranked blended 
learning as the most preferred delivery mode even though external and demographic elements 
have little impact on a student’s choice to take blended learning courses. Students were found to 
view their expected course grade or desire to participate in a course of low value in their course 
ratings. Instead, students viewed course objectives and progress toward the objectives as 
important, along with enjoying an effective learning environment and effective communication 
from the instructor (Dziuban et al., 2018). 


UI/UX considerations 


Definition 

The phrase “user experience” has permeated most industries in recent years. The demand for 
user experience (UX) designers and researchers continues to grow. Colleges and universities 
are taking notice, creating undergraduate and graduate degrees focused on UX offered in- 
person and online. 


Definitions of UX vary. The International Organization for Standardization (ISO) defines UX as 
“user’s perceptions and responses that result from the use and/or anticipated use of a system, 
product or service” (ISO FDIS 9241-210, 2009), whereas the definition by Norman and Nielsen 
(n.d.) explains that UX “encompasses all aspects of the end-user’s interactions with the company, 
its services, and its products.” Multiple researchers assert that there is no definitive definition of 
UX (Law, Roto, Vermeeren, Kort, & Hassenzahi, 2008; Bevan, 2008; Park, Han, Kim, Cho, & Park, 
2011). 


At its essence, UX is how a product responds when a user interacts with it, and how the user feels 
about that interaction. UX is determining how the product works on the outside, rather than on 
the inside (Garrett, 2002, p.10), with the ideal being a high level of ease of use for the user. 


Approaches 

Garrett (2002) provided a visual way to demonstrate how a website user experience is more than 
just the visual design. He asserted there are five planes tied to the user experience: Strategy 
Plane, Scope Plane, Structure Plane, Skeleton Plane, and Surface Plane. In addition, the elements 
are split in half, divided into “Web as a software interface,” which mainly addresses user tasks, 
and “Web as a hypertext system,” which deals with the information on the site (Garrett, 2002, 

pp. 31-33). 


Garrett (2002) explained that each plane is important for a combined positive user experience. 
Although the only plane the user sees is the Surface Plane, or the user interface (UI), all the other 
planes must support the planes above and beneath. If one plane is weak, the entire design could 
collapse in terms of UX. Garrett’s Elements of User Experience is widely used and cited as a 


time-tested approach for accomplishing user-centered design, as well as setting up the 
possibility of a positive user experience (Garrett, 2002). 


Morville (2004) offered an alternative to Garrett’s model: the user experience honeycomb, which 
declares that the user experience should be evaluated to make sure it is useful, usable, valuable, 
desirable, findable, credible, and accessible. Morville asserted that when designing a system, 
each of the honeycomb elements are crucial to providing a good user experience. As with 
Garrett, if one of the honeycomb components is lacking, the user experience suffers (2004). 


Regardless of the model used, following the method does not automatically equal a positive user 
experience. All components must be in harmony to achieve the optimal user experience. 


UX and UI Design 


Though UX design and UI design are often used together, and even seen as UX/UI design, these 
titles are not interchangeable. UI design focuses on the visual presentation and interactivity, as 
well as visually moving the user through the system (Lamprecht, 2019). Usability.gov (User 
Interface Design Basics, n.d.) contends that UI design joins the interaction design, visual design, 
and information architecture by anticipating what tasks users need to accomplish and ensuring 
the design elements are easy to understand and effectively aid users in those tasks. 


Chung (2014) defined the approach of UX design as user centered. UX design looks at the 
interaction between the user and the system, but it also encompasses research, prototyping, 
development, and testing of a system (Lamprecht, 2019). The visual/interface and functional 
designs of a system are considered parts of the overall user experience. 


UI Design Process. Nielsen and Molich (1989) asserted that there are “five golden rules” when 
approaching the UI design of a system: 


Knowing the users of the system is critical 

Users should be involved in the design 

The user interface must be taken into consideration 

Measurements need to be created and employed to determine the UI’s usability 
The design should be iterative to address and fix any usability issues 


These rules hold up even today in the world of UX design, extending from just the UI to include 
the entire user experience of a system. However, Chughtai, Zhang, and Craig (2016) stated that 
the UI of a system is often ignored for more focus on the architectural design. 


UX Design Process. Numerous UX design methods exist, including those that have been 
incorporated into the Agile and Lean project management processes (Mullins, 2015). At its core, 
any UX design process starts with the user. Cao and colleagues (2015) stated that there are three 
stages of UX design: research, design, and user testing, with all three focusing on user wants and 
needs. With each iteration, these three stages are often repeated (Cao, Gremillion, Zieba, Ellis, 
2015). These are complementary with Brown and Wyatt’s (2010) spaces for design thinking: 
inspiration, ideation, and implementation. 


UX design is meant to be an iterative process, with internal and/or external testing done as often 
as possible (Cao et al., 2015). The types of testing are discussed in a future section, UX and 
Usability Evaluation. 


Numerous UX methods can be included in the design process, but the following are common 
among many processes, as suggested by Lamprecht (2019), and Cao and colleagues (2015, pp. 
12-23): 


User research. User research can be accomplished in a variety of ways, including: 


e User interviews, which is a one-on-one interview with someone who is part of the target 

audience(s). 

Surveys and questionnaires, where questions are asked of representative users to 

uncover user needs and wants. 

Focus groups, which involves a facilitator walking a group of representative users 

through user research questions. 

Contextual inquiry, where a user researcher visits users in their own work or other 

environment. 

Card sorting, where a representative user is given topics, often on cards, and asked to 

organize the topics in categories that make sense to him/her. 
In addition to user research, understanding the stakeholders’ needs and business goals is 
essential (Moule, 2012). Cao and colleagues (2015) agree, and even recommend, that 
stakeholder interviews should be the first step in the research stage of UX design. Garrett (2012) 
supports this, as users’ needs, and site objectives comprise the Strategy Plane in his five planes 
of the user experience. 


Personas. Once user research is completed, UX designers can use the data to create personas. 
Cao and colleagues (2015) asserted that personas are “perhaps the most important document 
you'll create for analyzing users (p. 43).” Personas are fictitious users based on user data and 
include demographic and psychographic information. Designers also create personas to give a 
name and face to the actual users, as the components of a persona include a name, photo and 
possible behaviors, and goals and/or frustrations when faced with tasks (Moule, 2012; Cao et al., 
2015). 


Prototyping. According to Moule (2012), prototypes are used to bring design ideas to life while 
ensuring the user requirements are addressed. Creating a series of prototypes can also simulate 
the actions with a product that a user might take when trying to accomplish a task. This can help 
discover design issues early in the process, as well as refine the vision and scope of the system. 
Prototypes range from low-fidelity paper prototypes and wireframes to high-fidelity mockups, 
which give a visual representation of the system, and interactive hi-fidelity prototypes, which 
look and act like the actual system but without the necessary code or functionality (Moule, 2012). 


As mentioned earlier, UX design is an iterative process, and this rings true for prototypes. 


Testing with users between every iteration of prototype is essential to allow for possible 
adjustments needed for the next level of prototype (Cao et al., 2015). 


Testing. The ideal is for user testing to happen early and often during the UX design process, as 
mentioned by Cao et al. (2015) and Roscoe, Branaghan, Cooke, and Craig (2017). Cao et al. 


(2015) recommend the practice of iterating, testing, and implementing the user feedback, then 
starting the process again until the system is ready for development. Myriad methods for user 
testing exist, and the most common are discussed in the section UX and Usability Evaluation. 
Design Thinking. The phrase “design thinking” grew out of a merger between two firms, David 
Kelly Design and ID Two (Brown & Wyatt, 2010, p. 33). Brown and Kelly realized they were doing 
a different type of design for clients, which involved designing not just products but experiences 
for consumers. Kelly, who was the founder of the Hasso Plattner Institute of Design at Stanford 
University, found that when asked about design, he began using the word “thinking” with it, 
creating the phrase “design thinking’. Knemeyer (2015) maintains that UX and design thinking 
have shared roots, though the concept of UX precedes design thinking by a decade. Knemeyer 
even calls them “two sides of the same coin” (2015, p. 66). Brown and Wyatt (2010) define the 
design thinking process as a combination of inspiration, ideation, and implementations, and 
these three are overlapping spaces instead of individual steps. Cao and colleagues (2015) 
asserted that design thinking means that a product should be designed based on what the user 
wants and needs. This means the end user is at the center of the design, which is the essence of 
UX design. 


Usability 


Definition. 

Before UX became the primary phrase to explain the ease and satisfaction of use with a product, 
usability was king. Today, usability is one component of the overall user experience. 
Usability.gov describes usability as how usability a product is while a user interacts with it, while 
UX “focuses on having a deep understanding of users, what they need, what they value, their 
abilities, and also their limitations” (Usability Evaluation Basics, n.d.). 


Approaches 
Usability has a long research past, and with that, numerous definitions evolved. Nielsen (1995, p. 
26) defined good usability as possessing the following five characteristics: 


Learnability. The system should be easy to learn to facilitate efficient task completion for users. 
Efficiency. Once a user learns a system, he/she should be able to accomplish tasks with a high 
level of productivity. 

Memorability. The casual learner should be able to remember how to use the system after being 
away from it for a period. 

Errors. The system should have a low rate of errors, and catastrophic errors cannot happen. 
Satisfaction. A user should be satisfied (subjectively) with his/her experience after using the 
system. 


Dumas and Redish (1999) defined usability as being about a user using a product and being able 
to accomplish tasks easily and quickly. Dumas and Redish further defined with the following four 
points: 

1) Usability has a focus on the user 

2) Users use products to be productive in accomplishing tasks 

3) Users want to use accomplish tasks without excessive time delays 

4) Users define the ease of use of a product 


Krug (2014) writes that usability is accomplished when “a person of average (or even below 
average) ability and experience can figure out how to use the thing to accomplish something 
without it being more trouble than it’s worth” (p. 9). According to Krug, the first law of usability is 
“don’t make me think” (p. 11). 


Researchers and practitioners agree that the primary way to determine a product’s or system’s 
usability, as well as the design and entire user experience, is through testing (Krug, 2014; 
Chughtai et al., 2016; Roscoe, Allen, Weston, Crossley, & McNamara, 2014; Medina-Flores & 
Morales-Gamboa, 2015). 


UX and Usability Evaluation 


Definition 

At its core, testing for UX and usability involves evaluating a product with real users to obtain 
data about how users interact with interfaces (Nielsen, 1993, p. 165). UX evaluation has its roots 
in usability testing, sharing some techniques and adding new methods to address all elements of 
a user experience. Bevan (2009, p. 1) contends that evaluation for UX can be “interpreted as 
user-centered design methods for achieving user experience.” 


Some practitioners still refer to UX and usability evaluation collectively as usability testing 
(Unger & Chandler, 2012, p. 292), others have combined them under an umbrella called User 
Research (User Research Basics, n.d.), and others have them separated into different areas of 
evaluation (Bevan, 2009). No matter the phrase or school of thought employed, all involve testing 
with actual users using the product. 


Approaches 
The following methods are more commonly used when evaluating a system for usability and UX 
issues. 


Surveys and questionnaires. Surveys and questionnaires are used for users to give feedback 
on their experience using a product. Many studies performed that focus on UX and usability 
evaluation make use of surveys or questionnaires. While this approach does not directly observe 
the user, it can be used to extrapolate users’ perceptions and satisfaction with a system. 


A number of questionnaires exist to measure usability and UX, including the System Usability 
Scale (SUS), which is a well-used 10-question questionnaire with a Likert scale of 5, from strongly 
agree to strongly disagree (System Usability Scale, n.d.). Another popular questionnaire is the 
User Experience Questionnaire (UEQ), which provides questions that use Likert-scale 
measurements on attractiveness, perspicuity, efficiency, dependability, stimulation, and novelty 
(User Experience Questionnaire, n.d.). 


The challenges with using surveys and questionnaires include recruiting participants that will 
give an appropriate sample size, as well as ensuring questions are written in a way to obtain the 
data needed without leading the participant (Unger & Chandler, 2012, p. 108). Dumas and 
Redish (1994) agree that while surveys allow you to collect user preferences and attitudes, they 
do not allow you to observe the user interacting with the system. 


Focus groups. Focus groups bring together a group of target audience members to discuss 
aspects of a product or brand (Unger & Chandler, 2012, p. 121). Bevan (2009) considers focus 
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groups to be a method to gather user opinion. According to Dumas and Redish (1999, p. 45), 
conducting a successful focus group means making sure those invited to participate are 
representative users, the questions are planned and are written to obtain the needed 
information, and individual leading the discussion with the focus group must understand the 
goals of the testing. 


Challenges with focus groups is the idea of “group-think,” where the setting could influence 
some group members’ statements (Unger & Chandler, 2012, p. 121). In addition, Dumas and 
Redish (1999, p. 45) warn that while, you can gather user beliefs and attitudes, focus groups are 
not meant for discovering how a user uses the product. Lewis (2012, p. 1269) agrees that with 
focus groups, there is no observation of participants performing tasks. 


Heuristics. Heuristics are guidelines with which an evaluator rates a product. The evaluator 
likely is an expert in the subject matter, a UX professional, or someone close to the project, such 
as a developer. This is an internal evaluation, as actual users are not evaluators (Heuristic 
Evaluations and Expert Reviews, n.d.). Roscoe et al. (2018) asserted that heuristic evaluations are 
the most popular of the internal usability methods, and this method’s extensive use is supported 
by Magal-Royo and colleagues (2007) and De Lima Salgado and Freire (2014) (Magal-Royo, 
Peris-Fajarnés, Tortajada, & Defez, 2007). 


Heuristics have been created for specific systems such as software or e-commerce websites. 
Perhaps the most well-known heuristics are those of Nielsen and Molich (1989), which were 
refined and added to by Nielsen (1993, pp. 115-151): 


Visibility of system status 

Match between system and the real world 

User control and freedom 

Consistency and standards 

Error prevention 

Recognition rather than recall 

Flexibility and efficiency of use 

Aesthetic and minimalist design 

Help users recognize, diagnose, and recover from errors 
10) Help and documentation 


With heuristics, it is recommended that a severity scale be chosen to help categorize the order 
in which issues should be fixed. Various scales exist, including the 0-4 scale introduced by 
Nielsen (1993), with the severity increasing with the number, and a 1-4-point scale introduced by 
Dumas and Redish (1999), where the lower numbers are more severe. 


According to Nielsen (1993), a cognitive walk-through is a similar inspection method, as the 
participants in the walk-through often are usability experts or the individuals developing the 
system. The participants step through the system to catch any areas where the system is not 
performing as expected. Cognitive walk-throughs are considered internal testing, and 
researchers agree, including Chughtai, et al. (2016), Davids, Chikte, and Halperin (2013), and 
Roscoe et al. (2014). Roscoe et al. (2018) add that cognitive walk-throughs can help system 
designers and developers anticipate and plan for some user exploratory behavior. 


One of the main challenges with heuristic evaluations, including cognitive walk-throughs, is that 
actual users are not involved in the evaluation process. Nielsen (1993) agreed that with there 
being no real users involved, there is little chance to find users’ actual needs. 


Task-based usability testing. Often when the phrase “usability testing” is uttered, the 
reference is to task-based usability testing. Task-based testing involves creating scenarios and 
tasks for test participants to accomplish, or not accomplish if the usability is lacking. According 
to Nielsen (1993), scenarios are concise scenes created to help the participant better understand 
the context of the tasks that follow. Dumas and Redish (1999) have two takes on scenarios: a 
scenario could be a task with a sentence or two describing the context beforehand, or a scenario 
could be a short scene to help set the context of the related tasks that follow. Regardless of the 
definition, the one constant is the test participant, which should be a representative user of the 
system (Nielsen, 1993; Lewis, 2012). 


In addition to tasks, questionnaires or interviews can be done before and/or after the task 
portion. In general, questions asked before are to gather demographic and psychographic 
information, as well as possible participant knowledge of the subject matter or the system itself. 
Questions asked after can give additional information about what the participant felt about the 
test and the system after interacting with it. 


At least one facilitator will welcome the participant and describe the process, sometimes through 
a script created as part of the testing materials. Krug (2010) encourages the use of a script to be 
consistent in wording for each participant. Facilitators also are encouraged to get the participant 
to speak out loud. A think-aloud protocol often is used when participants perform the tasks 
(Dumas & Redish, 1999). Participants are encouraged to voice their thoughts, including any 
frustrations or unexpected issues they encounter. Krug (2010) asserted that the primary job of 
the facilitator is to keep the participant talking to get an understanding of what is confusing or 
frustrating. 


Challenges with task-based testing mainly point to budget. While performing usability testing is 
beneficial to any business, the company decision-makers may perceive testing as overly 
expensive. Nielsen (1993, p. 17) introduced the concept of “discount usability testing,” which 
involves utilizing task-based usability testing, along with thinking aloud and heuristics. Krug 
(2014, p. 116) offers a “do-it-yourself” option that takes minimal time and money if hiring a 
usability expert is not an option. 


Another challenge is finding representative users as participants. According to Nielsen (1993), 
the goal is getting participants that are as close to representative as possible. Krug (2014) agrees 
that representative users are ideal, but this may not be possible. There is a benefit from having 
not all participants be representative users; getting opinions from those who have not visited a 
specific website can provide good information, specifically about how new users might behave. 


Combining techniques. Magal-Royo et al. (2007) believe usability testing can be done at any 
stage, including requirements-gathering and development, and techniques can be combined. 
Nielsen’s (1993) ‘‘discount usability testing” attests to this, combining heuristic evaluations with 
task-based testing. The use of more than one approach can increase the likelihood to find more 
usability issues (Davids et al., 2013; Nielsen, 1993). 


UX and Usability Evaluations of Online Learning Systems 


Surveys and questionnaires 

Utilizing surveys and questionnaires appears to be the go-to method for evaluating the UX and 
usability of online systems such as websites, and this is equally true in evaluating online learning 
environments. Numerous studies exist in online learning, including the following studies. 


Sung and Mayer (2012 used a customized usability questionnaire that asked users to explain 
their experience with an e-learning system. The questionnaire focused on ease and satisfaction 
of use, as well as awareness and comprehension of the lesson material. Evaluating for 
navigational and signaling aids in the e-learning system, the authors uncovered that the mean 
ratings for the groups given navigational aids and signaling aids were significantly higher than 
the groups who did not have the aids. In addition, the effect sizes of lesson learning ranged from 
d=0.50 to d=1.35. 


To measure the UX on a Moodle-based online learning environment, Santoso and colleagues 
(2016) used an adapted version of the UEQ. The results from the questionnaire allowed the 
authors to see that while the pragmatic quality of the environment fared well, the hedonic quality 
was considered neutral. This enabled them to include UX improvements for future development 
plans (Santoso, Schrepp, Isal, Utomo, & Priyogi, 2016). 


Yulianandra, Wibirama, and Santosa (2017) utilized both the SUS and UEQ questionnaires to 
examine the relationship between task and website complexities ina web based LMS. The 
results revealed an inverted U relationship existed between the complexity of the LMS and 
students’ perceptions of the user experience and usability. They also posit that users will 
perceive an LMS to be less usable if the tasks are more difficult. 


In the realm of web-based multimedia in eLearning, Mackey, and Ho (2005) studied the 
correlation between the usability of web-based multimedia tutorials and students’ perceived 
learning from those tutorials. Specific usability factors were implemented in the tutorial design, 
including content, file size, response time of system, screen size of display, and user control. The 
authors used a custom survey instrument focused on the ease of use of the multimedia tutorials. 
The results of the study found a positive relationship between the presence of those usability 
factors and how favorable the students perceived their learning (Mackey & Ho, 2005) 


focus groups 
Few examples exist in the literature about the use of focus groups in online learning systems. 


Berking and Haag (2015) considered focus groups to create discussion in either a formal or 
informal learning experience. One example of use in research is from Roscoe et al. (2014). In 
their project to create a writing-based intelligent tutoring system (ITS) using usability testing and 
development, the authors created prototypes of the system and presented those to two focus 
groups, which consisted of high school English teachers. The results of those focus group 
sessions drove a redesign of the modules based on feedback about adding more interactivity 
and making it more engaging (Roscoe et al, 2014). 


Heuristics 

As mentioned earlier, heuristic evaluations conducted by experts are prevalent in reviewing 
online products such as websites. The same is true for online learning systems. Medina-Flores 
and Morales-Gamboa (2015) customized an instrument to evaluate the LMS for the University of 
Guadalajara. The instrument included Nielsen’s heuristics as well as ISO standards and 
recommendations, and it addressed eight attributes, including navigability, reliability, easiness, 
and design. Six experts experienced with the system reviewed the LMS. The researchers found 
that navigability was one of the highest rated attributes, whereas the system had a “serious 
usability problem with reliability” (Medina-Flores & Morales-Gamboa, 2015). 


Davids et al. (2013) performed a heuristic evaluation on a web-based multimedia application for 
medical students. Prior to the evaluation, the authors conducted task-based usability testing with 
students and made changes to the application based on those results. The heuristic evaluation 
was a follow-up to the user testing, and it utilized Nielsen’s heuristics with a severity scale from 1 
(cosmetic problems) to 4 (catastrophic problems). The results of the evaluation found 22 
usability problems, and 11 were determined to be serious. The evaluation did uncover six 
additional issues that the user testing did not, including unnecessary animations and text, and 
the font size being too small. Davids et al.’s study is a testament to combining usability 
techniques to find as many usability issues as possible (Davids et al., 2013). 


Task-hased usability testing 

While researchers and practitioners contend that user-involved evaluation is the key to 
determining the usability of a system, finding examples of task-based testing in online learning 
systems is a challenge. Much of the research concentrates on self-report, survey-based 
evaluations (Davids et al, 2013; Lim, Ayesh, Stacey, & Chee, 2013; Sung & Mayer, 2012; Santoso 
et. al., 2016), or heuristic evaluations (Roscoe et al., 2017; De Lima Salgado & Freire, 2014; 
Magal-Royo et. al., 2007). 


Chughtai et al. (2016) pointed out that the existing literature is lacking evaluations focusing on 
observing users to enhance interface design and usability of an ITS. This remains true for other 
eLearning systems. As of the writing of this review, no empirical evidence exists for using task- 
based testing as a usability method for online learning systems. 


Combining techniques 

One exemplary study in combining testing methods on educational systems is from Roscoe et al. 
(2014). The authors concentrated on designing a writing ITS utilizing usability testing and 
development. They approached the design of the first version of Writing Pal (W-Pal) from a user- 
centered approach. In addition, the authors combined usability techniques, such as focus groups 
with teachers and researchers performing cognitive walk-throughs. Finally, the authors 
performed a feasibility study, which focuses on whether a project or product is viable enough to 
keep proceeding (Evaluation and Assessment Capability (EAC), n.d., 2019). 


While a feasibility study is not a task-based usability test, it does involve testing the product with 
actual users. The six-month in-vivo test included 141 tenth graders and two high-school English 
teachers, with the students writing prompt-based essays at the start and end of the study. 
Surveys were included to gauge students’ perceptions of the lessons, games, and feedback. The 
system included mechanisms for both automated (algorithm-based) and human evaluations to 
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rate the quality of the pre- and post-study essays (n = 113 students). On a six-point grading 
scale, the automated results showed essay scores increased from an average of 2.3 (SD = 0.8) on 
the pre-study essay, to 2.8 (SD = 0.8) on the post-study essay. Human ratings increased as well, 
with the pre-study essays averaging 3.0 (SD = 0.6) and the post-study essays averaging 3.3 (SD = 
0.6). The results from all the usability testing drove the changes for the second version of W-Pal 
(Roscoe et al., 2014). 


Pragmatic/hedonic aspects of UX in online learning 


Definition 

Hassenzahl (2007) stated that people’s perception of interactions with UX has two different 
qualities including pragmatic and hedonic, also known as the pragmatic/hedonic model of UX. 
Pragmatic qualities refer to the functionality/utility and usability features of the product (2003). A 
pragmatic product is primarily instrumental, it can be used to achieve any given behavioral 
goal. For example, a cell phone is functional and usable because people know how to use it to 
make a call and send messages. Hedonic qualities underscore individuals' psychological needs, 
such as pleasure and interest. Products need to provide users with more than just pragmatic 
needs and suggests three attributes of hedonic qualities for experiences: stimulation, 
identification, and evocation. 


Products need to be stimulating for users, especially at first, so that they can generate a new 
impression or a sense of novelty. After securing attention, products need to resonate with users 
to some extent and be meaningful/relevant to them. The notion of relevance is identification. 
Lastly, users can be triggered by products in relation to their prior experiences or memories. 
For example, playing vintage video games can easily trigger a user’s good memories of the 
past. In short, pragmatic qualities ensure the effectiveness and efficiency of products, and forms 
the basic requirement of a product, while hedonic aspects ensure the satisfaction, pleasure, and 
appeal of products (Hassenzahl, 2003, 2007). 


Applications of Pragmatic/Hedonic Model of UX in Online Learning Context 
When UX was applied in the context of e-learning technologies and platforms (e.g., LMS), the 
hedonic factor receives more attention in the design of LMS because, as shown above, hedonic 


factors can enhance learners’ motivation and engagement by fostering their relatedness and 
autonomy over the learning (Zaharias & Pappas, 2016; Zaharias, 2009; Santoso et al., 2016). 


Zaharias and Pappas (2016) surveyed 808 learning professionals including learners (33%), LMS 
administrators (25%), and professors and trainers (42%). They found that usability in the e- 
learning context requires additional attributions for pragmatic quality. The traditional usability 
parameters are no longer enough to evaluate the effectiveness of e-learning courses or 
platforms (Zaharias, 2009). The design of learning experiences became the new focus instead of 
instructional design alone. 


Santoso and colleagues (2016) developed an adapted version of the user experience 
questionnaire (UEQ) containing six scales: attractiveness, efficiency, perspicuity, dependability, 
stimulation, and novelty; and use it to evaluate a student-centered Moodle-based LMS with 230 
computer science students. In particular, the pragmatic quality consisted of efficiency, 
perspicuity, and dependability, while the hedonic qualities included stimulation and novelty. 
Their findings from the UEQ showed that pragmatic qualities were captured as acceptable, but 
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hedonic qualities were found as neutral. Therefore, to capture the hedonic quality aspects of the 
online learning courses and platforms, more studies are needed in the future development of UX 
in online learning context (Santoso et al., 2016). 


UI and UX Design in Online Learning Systems 


Jearning experience design 

According to Hudson (2017), learning experience design (LXD) combines “elements of adult 
learning and development, instructional design, psychology, neuroscience, design thinking and 
UX design”. Benedek (2015) adds that LXD brings together UXD and interactive media design 
principles within an educational environment. 


Walcutt and Schatz (2019, p. 86) further defined LXD as involving the disciplines of industrial 
knowledge design (InKD), experience design, experiential learning, behavior economics, and 
human-systems integration (HSI). HSI focuses on a system's life cycle, and its core tenants are to 


Emphasize Humans: Human performance is the focus throughout the design process. 
Optimize the Total System: The system must be optimized as a whole, not as individual 
components. 

Consider the Full Lifecycle: The entire system must be maximized for its benefits while 
being cognizant of the risks and costs. 

Facilitate Design: Multidisciplinary design is emphasized, with stakeholders and 
designers working together (Walcutt & Schatz, 2019, p. 92). 


Design considerations. Garrett’s (2002) five-plane approach is widely used when approaching 
website and software design, and even has uses in the educational realm and LXD. Walcutt and 
Schatz (2019) contend that LXD can have a sizable and positive effect when using Garrett’s five 
planes or a similar goal-focused method. Benedek (2013) also references Garrett’s work when 
approaching LXD. 


Walcutt and Schatz (2019, p. 94) recommend the following when approaching LXD: 


Determine what the actual goal is and focus on it as well as the big picture. 

Use a holistic approach to user-centered design, including aesthetic design. 

Realize that humans have flaws and account for that in the design. 

Design using five facets of holistic experiential design: sense, feel, think, act, and relate. 
Utilize HSI methods to address scaling designs within larger organizations. 

Think of learning experiences as a collective set, rather than unconnected events. 
Consider LXD a synthesis of methods, theories, and tools from current and emerging 
disciplines. 


Hudson (2017) pointed out that a major challenge of LXD is that last point from Walcutt and 
Schatz (2019): finding the right blend of methods, theories, and tools for each educational 
environment. 


Blended Learning and UX 


Baehr (2012) notes that learners decide to use technology for several reasons such as its 
appropriateness and usefulness for their learning needs. Factors that influence their decisions 
are recommendations, richness, experience, and perception. Blended learning environments 
add both spatial and temporal components to an increased level of collaborative knowledge 
sharing. To optimize the learning experience, instructional designers and subject matter 
experts must follow design principles that support learners’ choices in media and tool selection 
while considering media richness factors. Designers and subject matter experts should also 
foster trust by personalizing student learning and should encourage collaborative activities 
inside and outside the training environment. 


Accessible Design 

Sanchez-Gordon and Lujan-Mora (2015) say that with usability, there is no single definition for 
accessible design on which researchers and practitioners can agree, though most definitions 
address the need for systems to be accessible to all, including those with disabilities. Disabilities 
can be personal or environmental. Personal disabilities include those associated with vision, 
hearing, speech, motor skills, cognitive, psychosocial, language barriers, and cultural 
considerations. Environmental disabilities, which are usually temporary, include external 
lighting or auditory conditions, internet access, and technology limitations (Sanchez-Gordon & 
Lujan-Mora, 2015). 


Creating an accessible online learning environment can be a challenge, but in the case of 
American public entities, it is required. Title III of the Americans with Disabilities Act of 1990 
specifically states, “title III covers access to Web sites of public accommodations” (2010). In 
addition, ADA and the Rehabilitation Act of 1973 state that any state or local government entity, 
including universities, that receive federal funding must abide by federal guidelines. 


Design considerations. The Web Content Accessibility Guidelines (2018) provides guidelines 
for designing accessible websites, including: 


Provide text alternatives for non-text content. 

Give alternatives to any time-based media, such as video or audio. 

Create content that can be adapted structurally to ensure information can be presented 
in more than one way. 

Ensure contrast between foreground and background element and provide audio 
controls on any audio longer than three seconds. 

Provide for functionality to be available from the keyboard, not just with a mouse. 

Allow for adjustable timing so users have enough to consume the content. 

Avoid designing content that could cause seizures, such as content that flashes more than 
three times in one second. 

Provide alternative ways to navigate to content. 

Ensure content is readable and written to be understandable. 

Design content that behaves consistently and predictably. 

Design error prevention functionality to assist users in correcting mistakes or avoid them 
altogether. 

Keep user agents such as assistive technology in mind when designing content. 


Universal Design 

According to the U.S. Department of Education, UDL is a framework for guiding educational 
practice which contains two main perspectives: (a) providing flexibility in the ways information 
is presented and learners are engaged; and (b) reducing barriers in instruction by providing 
proper supports and challenges, as well as maintaining high achievement expectations for all 
learners regardless of disability status (Patzer & Pinkwart, 2017). 


To summarize, three guidelines are emphasized within UDL: representation, expression/action, 
and engagement (Basham & Marino, 2013). Kang et al. (2018) further explained the three 
guidelines in detail. For instance, teachers should appropriately and strategically display the 
information by using different representations to help learners to solve the problem such as 
using analogy, multiple examples, and using multimedia. Then, students can fully execute their 
physical actions and communication to achieve their long-term goals, such as practicing with 
feedback until mastering the skills and solving the problems. Lastly, learners should be 
provided with multiple opportunities for engagement, which means the task should be 
reasonably challenging and not demotivate learners or make them feel bored and frustrated. In 
this manner, promoting interest and fostering self-regulation are the two main focuses (Kang et 
al., 2018). 


Applications of Universal Design in Online Learming Context 
Due to the fact that the majority of UDL research has been conducted in the K-12 education 
settings (Rao, Ok, & Bryant, 2014), Schreffler et al. (2019) conducted a systematic literature 
review on UDL in post-secondary STEM education for disabled learners. They pointed out that 
UDL is an effective approach to make STEM content accessible to all learners (Schreffler, 
Vasquez III, Chini, & James, 2019). Using UDL in higher education could increase the retention of 
the disabled learners and underrepresented populations (Newman et al., 2011; Rao et al., 2014). 


For instance, Street et al. (2012) examined the effect of Universal Design for Instruction (UDI) on 
Peer-Led Team Learning (PLTL), a national peer mentoring model designed to promote student 
success in STEM courses. However, the authors found that disabled students did not benefit from 
PLTL as much as non-disabled students. Then, they incorporated the principles of UDI into the 
PLTL and found out the overall positive trends in STEM persistence rate and the effective use of 
learning strategies among at-risk learners. They stated that STEM faculty need to be well-trained 
to properly utilize UDL and UDI to support at-risk learners’ STEM education (Street et al., 2012). 
Researchers also need to validate the effectiveness of the UDL principle in higher education 
STEM, due to a lack of research found for UDL in this area (Rao et al., 2014; Basham & Marino, 
2013). Especially, there is a need for investigating UDL-based inspired courses in postsecondary 
STEM education across several diagnoses including ADHD, learning disabilities, autism, and 
other similar diagnoses. 


Al-Azawei, Parslow, and Lundqvist (2017) examined the effectiveness of three main principles of 


UDL in an e-learning context by conducting a mixed research design combining survey and 
action methods with 92 undergraduate students. The three main principles being multiple means 
of representation, action and expression, and engagement. They found that traditional curricular 
limitations could be addressed by integrating UDL into instructional technologies due to the 


improvement of learner experience (e.g., perceived satisfaction and perceived usefulness) in e- 
learning setting (Al-Azawei, Parslow, & Lundqvist 2017). 


Kumar and Wideman (2014) conducted a case study of a technology-enhanced, in-class health 
sciences course with the adoption of UDL principles. They found that students’ overall 
perceptions in terms of this course were positive, resulting in an increase in satisfaction. This 
health sciences course was designed following the same three UDL principles, by incorporating 
multimodal means of representations, expression, and multiple means of engagement. The 
survey and interview results showed that flexibility, social presence, stress, and success are the 
four major themes that accounted for the positive perceptions of this course. Therefore, they 
suggested that instructors should make use of UDL-inspired courses to afford students with 
enough control over learning to reduce stress. This approach also could provide a high degree 
of social presence to increase the connectedness between learners and instructors, so that their 
level of confidence and engagement would be improved (Kuman & Wideman, 2014). 


Burgstahler (2002) discussed access, legal, and policy issues that would affect the accessibility of 
distance learning courses and presented an overview of design considerations for assuring that 
the distance learning courses are accessible to everyone. For instance, there are many access 
challenges faced by students and instructors in distance learning courses, including mobility 
impairments, visual/speech/hearing impairments, learning disabilities, and seizure disorders. 
The Americans with Disabilities Act (ADA) of 1990 required ‘‘people with disabilities have access 
to public programs and services, regardless of whether or not they are federally funded” (U.S. 
Department of Justice, 1990). The author further provided examples of policy considerations for 
making distance learning accessible to everyone including: (a) on-site instruction (e.g., 

facilities, furniture, restrooms, telephones and parking spaces are accessible to individuals with 
disabilities), (b) electronic communication (e.g., assistive technology), (c) web pages (e.g., 
accessibility test), (d) printed materials (e.g., tactile form for blind learners), (e) videotapes, 
video clips or televised presentations (e.g., provide captioning or transcription), and (f) audio 
conferencing (e.g., using public relay service for deaf students). Due to the rapid development 
of technologies, stakeholders, decision makers in the organizations, students, and instructors 
with disabilities should collaborate to ensure that accessibility policies, procedures, and 
guidelines are developed and implemented (Burgstahler, 2002). 


Elias (2010) identified and tailored a set of Universal Instructional Design (UID) principles 
appropriate to distance learning, so that it can meet the needs of e-learning instructional 
designers and instructors for improving students’ learning outcomes. The author then evaluated 
the effect of these modified UID through assessing the accessibility level of a sample UID- 
inspired Moodle course. In particular, the author proposed eight UID principles: equitable use, 
flexible use, simple and intuitive designs, perceptible information, tolerance for error, low 
physical and technical effort, community of learners and support, and instructional climate. The 
author suggested that addressing accessibility issues is the fundamental step for using Moodle 
platforms by educational institutions. Therefore, an adequate consideration for assistive 
technologies and appropriate pedagogical approaches that can remove barriers is critical to 
successfully effectively the affordances of the learning management system (Elias, 2010). 


mLearning and designing for mobile 

Krug (2010) contended that testing for mobile usability is like testing on any device, though the 
logistics are different. Due to the ubiquitousness of mobile devices, as well as the varying screen 
sizes and operating systems of those devices, testing needs to consider whose device will be 
used (facilitator’s or participant’s), how (or if) the participant will be holding the device, how the 
facilitator will be able to see the screen, and the possibilities and logistics of recording of the 
session (Krug, 2010; Kukulska-Hulme, 2007). 


Magal-Royo and colleagues (2007) suggested combining usability methods to evaluate an 
mbLearning system. They believe using heuristics evaluations, cognitive walk-throughs, and 
observation-based testing at various stages of the building, development, validation, and 
verification of the system will help maximize the system’s usability (Magal-Royo et al., 2007). 
This corresponds with the thinking of Davids et al. (2013) and Nielsen (1993) that a combination 
of testing methods will uncover the optimum level of usability issues. 


Designing for Educational Games 

Gamification is a trending topic in the realm of education (Kasurinen & Knutas, 2018; Alvarez- 
Xochihua, Merino, Organero, Kloos, & Gonzalez-Fraga, 2017). The term gamification broadly 

refers to the adoption of game-like elements in the design of educational systems to enhance 

users’ engagement and motivation to increase their retention rate (Kasurinen & Knutas, 2018). 


However, using games to increase motivation and retention rate is not an innovative concept in 
the other domains such as computational fields or business. For example, the serious games are 
considered a useful tool to enhance retention rate and participation (Gobel, Hardy, Wendel, 
Mehm, & Steinmetz, 2010; Estellés-Arolas, & Gonzalez-Ladrén-De-Guevara, 2012; Alario-Hoyos, 
Pérez-Sanagustin, Kloos, & Mufioz-Merino, 2014). Such serious games include games for health, 
crowdsourcing, and online education. Due to the similarity of gamification and other game- 
related concepts, such as playful design, serious gaming, and games for health, Deterding, 
Dixon, Khaled, and Nacke (2011) believed that it is important to differentiate the gamification 
concept from the concepts of playful design and serious games. 


Deterding et al. (2011) proposed the different solutions in terms of the definitions of the 
gamification and serious games. First, gamification (gameful design) is related to games (the 
characteristic for games), not “play” (playfulness). Similarly, the serious game is a form of 
interactive computer-based game software focused on interaction rather than entertainment. 
Secondly, gamification is the use of game design elements in non-game contexts which merely 
focuses on incorporating elements of game, while serious games implied the design of full- 
fledged games for non-entertainment purposes (Brathwaite & Schreiber, 2008). Lastly, 
gamification is a design, not the game-based technology or other game-related practices, while 
serious games are products with a “real” function and purpose (Deterding et al., 2011). 


The Application of Gamification in elearning Context 

Learning motivation is an important factor in the design of educational computer-based games. 
Serious games need to introduce some design principles such as usability and user experience. 
To try to understand which design criteria can make educational games more successful, 
Alvarez-Xochihua et al. (2017) conducted a study with 41 masters students to evaluate two 


competition-based educational computer games. They found that there was a strong positive 


correlation between usability and user experience with learning motivation. Similarly, Kiili and 
colleagues (2012) presented a flow framework based on flow theory and tested this framework 
in the RealGame business simulation game case study. They found that the sense of control, clear 
goals, and challenge-skill dimensions scored highest among all flow experiences. They 
concluded that flow framework is a useful tool in studying game-based learning experiences to 
design engaging game elements for educational games. Specifically, educational games should 
expand the player’s mind by presenting appropriate challenges to overcome (Kiili, de Freitas, 
Arnab, & Lainema, 2012). 


Blended Learning and Gamification 

Student to student interactions (resulting in increased classroom engagement) in hybrid learning 
environments that utilize game-based designs were found to mimic student to world interactions, 
except that the student to world interactions were more positive and intense in nature 
(Chritopoulos, Conrad, & Shukla, 2018). Klemki and colleagues suggest gamification has been a 
way to ‘flip’ a MOOC with the intent to make MOOCs more engaging with the aim of lowering 
attrition. Game elements are also valuable for visualization of learning using analytics. They 
suggest that designers carefully choose game elements with consideration of the learning 
scenario which should consider the specific learning population, the topic, the domain, and the 
problem to be solved using the game. Learning analytics can help relate the learner activity with 
personalized data which helps students unlock the full potential of educational games (Klemke, 
Eradze, & Antonaci, 2018). 


UX and At-Scale Systems 


UX is extremely important in learning at scale systems, in which the learners vastly outnumber 
the instructors in their ratio, especially as the learners’ attitude toward the system will impact 
their use of it (Nakamura, Marques, Rivero, de Oliveira, & Conte, 2019). However, the field of UX 
is still a growing one, and attempting to catch up to online resources that have been in 
implementation for years, such as online library services (Pennington, 2015). The gap of 
understanding the importance of UX and successfully implementing UX into the available 
systems has not gone unnoticed by those in the field, as very few studies have evaluated the UX 
of at scale systems (Nakamura et al., 2019). During the research of this section, only one study 
could be found that explicitly looked to compare two online learning systems, Moodle and 
iQualify, using a UX perspective for their evaluation (Nichols, 2016). The lack of research in this 
area should be noted, as while researchers agree on the importance of UX, they also seem to 
agree on its status of being under researched. 


Overall UT Design Goals for E-learning Systems 

Whether approach is LXD or universal design, or the environment is mobile or educational 
games, special considerations must be made for online learning systems when it comes to UI 
design. Unger and Chandler (2012) asserted that an eLearning system is task-based, and users 
should be able to easily follow the content. In addition, content should be organized in digestible 
chunks to enhance comprehension. Davids et al. (2013) believed that a poorly designed e- 
learning UI adds to the user’s cognitive load, thus hindering the ability to concentrate on the 
content itself. 
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Appendix C: Distributed Online Pedagogy Review 


So far in this report, we reviewed the state-of-the-art for designing blended, at-scale, and 
other distributed learning environments at the institutional and courseware levels. While these 
are critical aspects of distributed learning ecosystems, they do not address the fine-grained 
details, such as lesson and course design. This section discusses how to design effective 
distributed learning at the fine-grained level with a focus on designing effective lessons and 
courses. 


State-of-the-art distributed learning environments support communities of inquiry. 
State-of-the-art distributed learning environments support learner’s motivation 
State-of-the-art distributed learning environments design instruction aligned with 
human cognitive processing 

State-of-the-art distributed learning environments support self-regulation within their 
learners. 

State-of-the-art distributed learning environments provide appropriate guidance 
during instruction 

State-of-the-art distributed learning environments facilitate learning with retrieval 
practice strategies 

State-of-the-art distributed learning environments should use competency-based 
learning when the context is appropriate 


Social Theories 
Community of Inquiry 


In its inception, the Community of Inquiry (Col) framework, which is crafted around the Practical 
Inquiry Model, was posited as a robust tool to support online educational experiences in 
Computer-Mediated Communication (CMC) (Garrison, Anderson, & Archer, 2000; Garrison, 
Anderson, & Archer, 2001). A Colis a group of participants who intentionally collaborate 
through a cycle of critical thinking, discourse, and reflection to arrive at personal meaning and 
derive mutual understanding (Garrison, 2017; Rovai, 2002a; Shahrtash, 2017; Thompson & 
McDonald, 2005). The goal of the Col framework is to engage in deep learning through a 
collaborative, constructivist online community (Garrison, 2017; Garrison et al., 2001; Rovai, 
2002b; Thompson & McDonald, 2005). The construction of new knowledge is dependent upon 
prior knowledge and experience, meta-cognitive processes, and learning context (Kovanovic et 
al., 2018). Proponents of the Col point out that its constructivist emphasis does not preclude 
learners from achieving objectifiable learning outcomes, but that it is mainly concerned with the 
“educational transaction” of how learners construct knowledge (Akyol, Garrison, & Ozden, 
2009). Traditionally, in higher education environments, a sense of community has been 
associated with higher levels of learning due to collaboration and discourse (Garrison & 
Arbaugh, 2007). Three aspects to meaningful online learning, deemed elements, are identified 
as being requirements in the learning process, namely, cognitive presence, social presence, 
and teaching presence (Garrison et al., 2000; Garrison et al., 2001). Presence is the perception of 
having an interaction or active participation in an activity (McKerlich, Riis, Anderson, & Eastman, 
2011). These three presences are interwoven to maximize the success of higher educational 
experiences with online or telecollaboration at the core (Anderson, Rourke, Garrison, & Archer, 
2001; Garrison et al., 2000; Garrison, Cleveland-Innes, & Fung, 2010; Redmond & Lock, 2006; 
Tolu, 2013). 


It should be noted that there are critics of the Col (Annand, 2011; Maddrell, Morrison, & Watson, 
2017; Rourke & Kanuka, 2009). These researchers question the value of the Col framework as it 
has no supporting empirical studies that measure learning outcomes (Maddrell et al., 2017; 
Rourke & Kanuka, 2009) or studies to support the presumed knowledge co-construction that are 
not learner-perception derived. Skeptics of the Col framework question its current popularity for 
these reasons (Annand, 2011). 


Cognitive presence 

Cognitive presence, which is the ability of the individual learner to construct meaning through 
the online discourse, is central to the Col framework because students are known to construct 
meaning through critical thinking (Garrison et al., 2001). According to Garrison and Arbaugh 
(2007), cognitive presence in online formats is the most formidable presence to study and 
develop, a reflection that is supported by other research teams (Duphorne & Gunawardena, 
2005). Critical thinking is nurtured through sustained communication between the community 
stakeholders (Garrison et al., 2000; Gunawardena, Lowe, & Anderson, 1997). Hosler and Arends 
(2012) found that students’ perceptions of cognitive engagement did not differ between face-to- 
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face courses and online courses. In this framework, Garrison and colleagues found that students 
under the direction of the instructor (teaching presence), and the influence of the community 
members (social presence), move through a practical inquiry process. This process propels 
them from a triggering event that initiates learning through an exploration phase which allows 
the learner to deliberate on the applicability of the lesson. After deliberation, students move into 
a self-reflective integration phase which, ideally, leads to a resolution phase. The net result of the 
community interaction is to bring about a specific action or change in practice (Garrison et al., 
2000; Garrison et al., 2001; Garrison et al., 2007; Maddrell et al., 2017). Garrison and colleagues 
(2001) pointed out that the concept of cognitive presence is to be seen as a focus on higher- 
order thinking, rather than as a tool to promote specific and individual learning outcomes, and 
that critical thinking is, in itself both a process and an outcome (Akyol et al., 2009; Garrison, 
Anderson, & Archer, 2010). Arbaugh, Banjert, and Cleveland-Innes (2010) added that the Col 
may be more suited to applied disciplines rather than pure disciplines, as students’ perceptions 
of the three presences vary by discipline. 


The triggering event, which is brought on by the cognitive dissonance of having existing beliefs 
incongruent with new information, ushers the student into the inquiry cycle (Dempsey & Zhang, 
2019). In this phase of critical thinking, a student is beset by a sense of puzzlement, problem 
recognition, or questioning which evokes an inquiry (Garrison et al., 2001). This trigger can be 
due to the direct intervention of the instructor, which is likely in purely educational settings, or 
may be evoked by any Col member, which is likely in nonhierarchical instances of computer 
conferencing (Garrison et al., 2001). 


After the triggering event, students begin the exploration phase of learning, in which they begin 


searching for information that is relevant to solving the problem and sense-making (Dempsey & 
Zhang, 2019; Garrison et al., 2001). The third phase of the Practical Inquiry Model is the 
integration phase of learning which takes the student from the contemplation of problem-solving 
schemes into arranging the schemes into a new order for use in solving the problem (Dempsey 
& Zhang, 2019; Garrison et al., 2001). 


The final phase in the Practical Inquiry Model is the resolution phase, which ushers the student 
into the consideration of and commitment to solutions to the problem. Furthermore, problem 
resolution strategies stimulate deductive reasoning to test the validity of the solution (Garrison et 
al., 2001). This is a time for students to critically evaluate the chosen solution to detect reasoning 
errors and, if none are found, proceed on to anew problem (Garrison et al., 2001; Gunbatar & 
Guyer, 2017). It is noteworthy that, in practice, students rarely arrive at the resolution phase of 
learning and few come to experience the higher levels of integration (Arnold & Ducate, 2006; 
Dempsey & Zhang, 2019; Garrison et al., 2001; Vaughan & Garrison, 2005). Vaughan and 
Garrison (2005) suggested that topic selection could play a role in the lack of progression to 
resolution. Garrison et al. (2007) noted that a lack of goal sharing in collaboration may 
underestimate instances of resolution in transcript evaluation, which may account for low 
recorded instances of resolution. Garrison et al. (2007) called for instructors to add important 
discussion dimensions, such as direction and facilitation to online threads to foster resolution. 
Akyol, Vaughan, and Garrison (2011) found that students enrolled in longer courses had greater 
percentages of students arriving at the integration and resolution phases compared to those in 
shorter courses. Furthermore, group dynamics and functioning may play a role in how quickly 
students can move through the cycle of inquiry (Garrison et al., 2007; Tuckman & Jensen, 1977). 
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One obstacle in implementing the Col framework is the difficulty of moving students through the 
cognitive process from the triggering event to resolution (Garrison et al., 2001; Garrison et al., 
2007). When evaluating the practical inquiry model through the lens of online communications, 
research shows that students spend the majority of their time in the exploration phase of the 
cycle, which is predominantly informational exchange (Celetin, 2007; Fahy, Crawford, & Ally, 
2001; Garrison et al., 2001; Gunawardena et al., 1997; Kanuka & Anderson, 1998; Luebeck & Bice, 
2005; McKin, Harmon, Evans, & Jones, 2002; Meyer, 2003). Meyer (2003) suggested that students’ 
may be inadequately prepared to comment on the complexity of the issues raised in the online 
forum, so the learning cycle does not progress, or the course faculty may have overlooked an 
opportunity to take the discussion to resolution. Marton and Salj6 (1976) demonstrated that 
students have inter-group differences in qualitative learning and processing which result in 
differing demonstrable outcomes, which can aid in explaining different perceptions of cognition. 


The idea of cognitive presence is conceptualized as a continuum between the private world and 
the shared world of the students and instructor (Garrison et al., 2001). For example, for a student 
to enter the learning cycle, they must perceive shared information as applicable to them which 
moves the new information from the shared to the private world. In the private world, reflection 
on the content can trigger ideas that lead to discourse in the shared world. This discourse can 
generate an action step to solve the current problem, thus ending the cycle, or the discourse can 
generate new avenues to explore (Garrison et al., 2001). 


Social Presence 

Online collaborative learning is a shared experience between students themselves and between 
students and instructors in which affective and expressive communication is pivotal to learning 
(Dempsey & Zhang, 2019). Social presence in online learning is an attribute with which students 
make themselves “real” to the rest of the cohort for the purpose of fostering emotional (affective) 
expression, open communication, and group cohesion, although it also plays the practical role of 
attaining the goals of student fulfillment and student retention (Garrison et al., 2000). In online 
learning, factors that normally mediate social presence, such as facial expression, the direction 
of gaze, posture, dress, non-verbal cues, and vocal cues, are inherently missing from 
communication (Seckman, 2018; Tu, 2000). From the inception of the Col framework, Garrison et 
al. (2000) have insisted that social presence has less to do with the medium of engagement 
(online versus face-to-face) and more to do with the context of communication that is established 
through skills, learner motivation, organizational commitment, activity choice, familiarity with 
other participants, and the length of time using the learning medium. Social presence is 
progressive in the sense that students entering the online community first find an identification 
with the group that matures to trusting, purposeful communication, and then develops into social 
relationships (Annand, 2011; Garrison et al., 2010). 


Garrison et al. (2000) and Gunawardena (1995) argued that cognitive presence is more easily 
sustained for learners when the social presence is in place. However, Nagle and Kotze (2010) 
stated that social presence can develop because of the other two presences, but that is not likely 
to be a precursor to cognitive presence. Social presence is strongly and positively correlated 
with a student’s perception of learning quality, and a student’s perception of social presence is 
positively and significantly correlated to performance on a written assignment, but not to 


performance on examinations (Picciano, 2002). High student online interaction is related to 
higher scores on written assignments, but not to improved test averages (Picciano, 2002). 


The climate of online interaction should be one that allows for safe and comfortable 
communication for all participants through mutual and respectful discourse (Gunbater et al., 
2017; Tolu, 2013). Open communication is a result of experiencing a trusting environment 
(Dempsey & Zhang, 2019) and is a requirement, along with group cohesion, for a productive 
group inquiry (Garrison et al., 2007). Akyol et al. (2011) found that course length did not affect 
the frequency of open communication events. 


Affective expression is the sense in which an individual is accepted as a part of the group 
(Dempsey & Zhang, 2019; Garrison et al., 2007). Longer course duration has been found to 
significantly increase the frequency affective communication compared to shorter-term courses 
(Akyol et al., 2011). However, affective communication, along with instances of open 
communication, decreases over the length of the course, while group cohesion increases over 
time (Akyol et al., 2008; Vaughan & Garrison, 2006). 


Group cohesion is the successful creation of group identity within the Col (Tolu, 2013). Studies 
indicate that group cohesion in established groups is significantly more influential on social 
presence and task participation than the media condition (audio conferencing versus video 
conferencing) (Yoo and Alavi, 2001). 


Teaching Presence 

Teaching presence, the third element of the Col framework, puts instructors in a dual-faceted 
role of designing, selecting, organizing, and presenting material, while also facilitating the 
computer-mediated learning (Garrison et al., 2000). When computer conferencing fails as a 
communication tool in online courses, it is perceived to be due to a lack of teaching presence 
(Gunawardena, 1991), as a teaching presence is the pivotal element that establishes and 
facilitates social presence and cognitive presence (Nagle & Kotze, 2010; Shea et al., 2014; Tolu, 
2013). The research consensus is that teaching presence is significantly and positively related to 
student satisfaction, perceived learning by students, and a sense of community in online courses 
(Garrison et al., 2007; Garrison et al., 2010; Kanuka et al., 2007; Swan et al., 2008; Tolu, 2013). 
Instructor scaffolding for interaction in online environments has been hailed as the most 
significant modifier for students’ self-regulation interactions with other community members 
(Cho & Kim, 2013). Furthermore, it is hailed as a predictor for student cognitive presence in 
online learning communities (Seckman, 2018) and, at a minimum, related to cognitive presence 
(Akyol et al., 2008). 


The design process for instructors wishing to utilize the Col can trace its history to the idea that 
learning that is structured around participation activities results in the construction and 
acquisition of knowledge (Gutierrez-Santiuste, Rodriguez-Sabiote, & Gallego-Arrufat, 2015). The 
aim of the chosen instructional design is to create and facilitate an environment for the 
construction of meaning by bringing together student motivation, interest, commitment, and 
learning (Gutierrez-Santiuste et al, 2015). This requires teachers to tailor learning to the student 
in a holistic manner, rather than simply focus on cognitive engagement. 


Facilitating discourse is one of the primary goals of the Col and is accomplished by motivating 
learners to engage and participate in discussion forums through modeling the discussion 


process and content, which will guide learners through the process of higher-order thinking to 
achieve learning (Tolu, 2013). Facilitation of discourse in the Col framework is not the sole job of 
the teacher, although the teacher assumes the role of assessing the discussion threads and 
encouraging content-focused responses (Anderson et al., 2001). Teacher facilitation in the Col 
discourse is the key component in promoting critical thinking (cognitive presence) (Hosler & 
Arends, 2012). This makes a compelling argument for instructors to model purposeful direction 
in threaded in online discussion forums that nudge students to engage in cognitive processes 
while constructing their replies, thereby stimulating critical thinking. 


Direct instruction is the mechanism teachers use to share subject matter expertise and provide 
organization in learning (Anderson et al., 2001). Students and teachers both expect the teacher 
to communicate in-depth content knowledge with excitement and interest in a pedagogically 
sound framework (Anderson et al., 2001). Lack of direct instruction can result in a lack of 
refinement of online postings and may be a cause of the failure of students to cognitively move 
past the lower phases of cognitive interactions (Anderson et al., 2001; Garrison et al., 2001). 
Instructors must utilize direct instruction, where necessary to foster student cognitive presence. 


Validity and reliability of the Col framework 

The Col framework is useful for the design and implementation of an online course (Garrison et 
al., 2007), but it is also a theoretical framework for research in student perceptions of online 
courses. Other common areas of research on the Col include evaluating the interrelationships of 
the proposed elements and verifying its theoretical frameworks (Garrison et al., 2007). Due to its 
use in both theory and practice, the instrument to evaluate the Col must, itself, be evaluated for 
reliability and validity. Support for the construct validity of the Col instrument has been 
established by many researchers (Arbaugh et al., 2008; Diaz, Swan, Ice & Kupczynski, 2010: 
Kozan & Richardson, 2014; Stenbom, 2018; Swan et al., 2008). Some have upheld the construct 
validity of the three-factor model but have suggested a modification of the present Col which 
bifurcates teaching presence into a design and organizational component and an instructor 
behavior component (Arbaugh et al., 2008; Diaz et al., 2010). Caskurlu (2018) has shown that the 
construct validity of the original three-factor Col framework has held up through confirmatory 
factor analysis. Reliability has been established for the survey instrument used in the Col to 
demonstrate social, teaching, and cognitive presence (Stenbom, 2018; Swan et al., 2008). 


New findings in the Col model 


Modification of social presence 

Social presence is questioned by some as not being impactive on cognitive presence ina 
tangible way (Annand, 2011). Students in a study by Ke (2010) found that adult students 
perceived their online relationships were a “bonus” of the online education process but were 
not required for learning. Shea and Bidjerano (2009b) reported that when students achieve 
integration and resolution in their cognitive processes, it may be due to the activities they 
completed rather than their participation in online forums. 


Covariates of gender, age, ethnicity, course discipline, online course experience, academic level 

Gorsky, Caspi, and Smidt (2007) found that, in adult distance education, students tend to study 
alone unless they encounter problems. Difficulties lead students to seek help from others 
(Gorsky et al., 2007). Furthermore, these authors found no relationship of this pattern to age, 


gender, prior acquaintance with another learner, motivation to achieve, or learning style. Shea 
et al. (2009a) found no relationship between age, gender, or academic level and any of the 
presences. Cho and Kim (2013) found that age, gender, online course experience, and the 
perceived importance of interaction with the instructor had no significant effects on the student’s 
self-regulatory mechanisms in online learning environments; however, the grade in school did 
increase the likelihood of students self-regulating for interaction in the online format. 


Relationships between presences 

As the exponential growth of digital technologies has ensued, the social aspects of teaching have 
come to bear on online interactions in new ways that blend the social and teaching presences 
(Armellini & De Stefani, 2016). For example, instructors usually answer class questions through 
email, which can elicit a less-formal approach to instruction (Armellini & De Stefani, 2016). 
Students perceive that the “social aspects” of teaching outrank peer interaction in the modern 
Col (Swan & Shih, 2005). 


The cognitive presence is also affected by social presence as students and instructors interact in 
social formats for clarification of learning (Armellini & De Stefani, 2016). It has been suggested 
by Armellini and De Stefani (2016) that both the cognitive and teaching presences be extensions 
of social presence since social presence encompasses interactions for learning, socialization of 
the content, the community of inquiry development, course design, self-study, and the learner’s 
experience. Cognitive and social presence interact at the level of the common purpose of the 
group and the need for social presence may diminish when there is no need for collaborative 
assignments (Picciano, 2002). Garrison et al. (2007) suggested that social presence provides the 
context for the high-level discourse into which teaching presence brings structure and 


organization to the course interactions, which encourages cognitive presence. Shea et al. (2014) 
proposed a revised Col framework that retains the organization; however, the social presence is 
seen as ancillary to each of the presences, thereby restructuring the framework into an 
interactive social learning presence, social teaching presence, and social cognitive presence. 


Some research has hinted at the need to modify the original Col to include a learning presence 
element to account for unique learner behaviors that are brought to every online course (Kang, 
Liew, Kim, & Park, 2014; Pool, Reitsma, & van den Berg, 2017; Shea & Bidjerano, 2010; Shea et al., 
2012; Shea et al., 2014; Wang & Kang, 2006). Specifically, learning presence accounts for the 
unique aspects of accountability, motivation, and strategies that each learner brings to the online 
learning opportunity. The learning presence is more related to course grades than are any of the 
other Col elements (Shea et al., 2012). 


Kilis and Yildirim (2018) suggested that regulatory presence is a more thorough treatment of the 
presence students bring into the learning process (versus learning presence) because students 
bring more than just behavioral variances into online learning. Regulatory presence is the 
learner’s unique combination of forethought, performance (volition), and reflection. Specifically, 
the cognitive presence is most strongly predicted by self-regulation and that teaching presence 
is most valuable to teaching presence, although metacognition and motivation are valuable. Self- 
regulation, metacognition, and motivation are shown to significantly contribute to the formation 
of the Col in online classes (Kilis and Yildirim, 2018). Individual metacognition may be 
insufficient to account for the group dynamics in collaborative interactions; shared 


metacognition and co-regulation may more accurately reflect individual and shared regulation 
in these settings (Garrison et al., 2015). 


Majeski, Stover, and Valais (2018) stated that emotional presence in learning exceeds the 
confines of emotional expression only and should include learner motivational, affective, and 
experiential elements. These elements include self-efficacy, self-awareness, openness, 
receptivity to others, and the management of arousal states common in learning and education. 
In this model, Majeski et al. (2018) adjusted the teaching presence of Col to include emotional 
presence and instructional presence which, along with learner presence, is defined as the 
learners’ experience. 


Xie and colleagues (2017) have research focused on establishing and maintaining a social 
presence in online learning environments; however, not all online social interactions are 
positive. They found empirical data showing conflictual presence is inherent in the social 
learning context. Student discourse is composed of learners establishing an identity for 
themselves as a certain type of facilitator or participant in the online discourse and then 
negotiating another identity to the group which is co-constructed in the discourse with other 
group members. When the identity and judgment ofa participant are questioned, the tension in 
the group escalates and group structure is challenged. Participants ascribe identities to 
themselves and others throughout the lifetime of the group (Xie, Lu, Cheng, & Izmirli, 2017). 


Open versus guided inquiry 

Gunbater and colleagues (2017) have some evidence that the inquiry type (open versus guided) 
plays a role in the students’ perceptions of teaching presence and cognitive presence. Students 
using guided inquiry ranked all presences higher than students in open inquiry settings leading 
researchers to claim that guided inquiry supports a more efficient working process than does 
open inquiry. Social presence scores were not significantly different between the two inquiry 


types. 


Deep and meaningful learning 

While the creation of online communities and the nature of online interactions have been well 
documented, there is less evidence to support the quality of the learning in the Col (Akyol & 
Garrison, 2011; Rourke & Kanuka, 2009; Maddrell et al., 2017; van der Merwe, 2014). Akyol and 
Garrison (2011) found support for the framework as a method of eliciting both perceived and 
actual learning outcomes. However, Maddrell et al. (2017) found no relationship between 
learning outcomes and students’ perceptions of Col participation. Rourke and Kanuka (2009) 
remark that students’ perceptions of the three presences stem from the learners associating the 
surface learning in the Col that takes place in independent activities and direct instruction. 
These authors stated that the perception of presence is not generated from the sustained 
discourse that characterizes the Col. 


Typical use cases in Col. 


Massive online open courses 

Kovanovic et al. (2018) investigated the Col framework in the context of the Massive Open 
Online Courses (MOOCs) and validated the survey instrument as capturing the original three 
presences as presented by Garrison et al. (2000, 2001). However, these authors found that the 
framework was better supported by the addition of three more factors, namely, course 


organization and design (a subfactor of teaching presence), group affectivity (a subfactor of 
social presence), and the resolution phase of inquiry learning (a subfactor of cognitive 
presence). The discovery of these additional factors highlights the differing dynamics between 
student perceptions of traditional online courses and MOOCs (Kovanovic et al., 2018). 
Furthermore, Kovanovic et al. (2018) underscored that the open nature of MOOCs, their limited 
teacher interaction, and short duration (compared to traditional online courses) negatively affect 
the participants’ ability to achieve the highest levels of cognitive presence. Also, the large 
student number and limited student-instructor interaction require instructors to conscientiously 
build strong organizational and design principles into MOOC courses. The most challenging 
aspect of using the Col framework in a MOOC environment is generating an affective bond in 
the student cohort, likely due to the short duration of the course and the large student group, as 
these factors run contrary to a strong social presence (Akyol & Garrison, 2008; Akyol, Vaughan, 
& Garrison, 2011; Kovanovic et al., 2018). Recently, Amemado and Manca (2017) have suggested 
a combined paradigm of the distributed learning approach applied to the Col in the cases of 
MOOCs. In this paradigm, the elements of the Col shift to distributed social presence, 
distributed cognitive and metacognitive presence, and distributed teaching presence and 
combines the constructivist approach of the Col with the connectivist approach of distributed 
learning (Amemado et al., 2008). 


Synchronous and asynchronous online instruction 
In a summary paper on improving student social presence in online classes, Newberry (2001) 


suggested that media richness played a role in fostering person-to-person connectedness. 
Rockinson-Szapkiw (2012) found that students in a class utilizing combined asynchronous and 
synchronous CMC did not have greater presence scores for teaching, social, or cognitive 
presence than students who learned only through asynchronous means. However, later research 
by Rockinson-Szapkiw (2015) found that students using synchronous or a combination of 
synchronous and asynchronous interaction scored higher in the Col survey than did the students 
participating with only asynchronous CMC. A study of nursing students revealed that the 
students learning in a synchronous environment outscored the students learning in an 
asynchronous environment in course engagement (Claman, 2015). 


Blended Learning in the Col 

Students in blended learning environments report higher teaching, social, and cognitive 
presence than those in fully online course environments (Akyol et al., 2009; Shea & Bidjerano, 
2012). This implies that face-to-face interactions strengthen the three presences and helps to 
make a more functional Col (Shea & Bidjerano, 2011; Shea et al., 2012). Akyol and colleagues 
(2009) found that the development of the three presence indicators was similar in both learning 
formats, but that when considering social presence in blended learning environments, group 


cohesion discourse was more frequent in the online-only format. However, the reverse was true 
regarding affective discourse. In the area of teaching presence, instructors in blended courses 
were more apt to use direct instruction and facilitating discourse compared to online-only 
formats (Akyol et al., 2009). Shea and colleagues have found that students in blended learning 
courses and students with previous experience in fully online courses scored higher on 
cognitive presence compared to students who were less familiar with the online delivery format. 
However, the difference in social presence and teaching presence between the formats 
accounted for the cognitive differences. Student self-regulated learning is predictive of cognitive 
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presence when course delivery and prior online course experience are controlled variables 
(Shea et al., 2012). 


Best Practices in a Col 

The research base for the CoI model is vast; however, several themes dominate the Col 
literature. For example, the valid and reliable Col measurement instrument is designed to assess 
student perceptions of their engagement in the meaningful learning process (Arbaugh et al., 
2008; Garrison et al., 2000; Garrison et al., 2001; Kozan & Richardson, 2014; Stenbom, 2018; Swan 
et al., 2008), not to measure deep meaningful learning via outcome measures (Akyol et al., 
2009). Researchers point out that references to deep learning should be taken in the context of 
advice on a practical approach to the design and implementation of the learning transaction 
(Akyol et al., 2009). One core belief of the proponents of the Col is that knowledge is constructed 
in online environments when the instructors and students establish social and teaching 
presences that facilitate epistemic engagement in reflection and dialogue (Shea et al., 2009). 
This engagement fosters cognitive presence and impacts the teaching methods from design to 
implementation (Shea et al., 2009). Deep learning depends on focused and united groups that 
can maintain quality engagement as they move through the learning process to the resolution 
phase (Akyol et al., 2011). Blended learning can produce a faster development of group 
cohesion and higher student perceptions of social, teaching, and cognitive presence (Akyol et 
al., 2009). 


Social presence is commonly displayed as a positive aspect of group interaction; however, Xie et 
al. (2017) describe a conflictual element in social interaction that may further illuminate the 
concept of social presence in the Col. Social presence is depicted by some researchers as a 
common component of a modified Col which includes learning, teaching, and cognitive 
presences (Shea et al., 2014) or as a fourth presence in the Col (Shea & Bidjerano, 2010). Some 
researchers argue that social presence is unimpactful in teaching or cognitive presence 
(Annand, 2011). Teaching presence, however, is seen as promoting cognitive presence (Hosler 
& Arends, 2012). 


Motivational Theories 

There is a substantial volume of literature on motivation in education. As an interdisciplinary and 
well-researched theoretical perspective, academic motivation primarily encompasses five 
contemporary theories, which are the expectancy-value theory, attribution theory, the self- 
efficacy theory, the achievement goal theory, and the self-determination theory (Aguilar, 2016; 
Cook & Artino, 2016). Each of these theories, while focusing on one facet of motivation, leaves 
out other important considerations, and many of the arguments overlap in their coverage of 
human motivation (Cook & Artino, 2016). Taken as a whole, though, these five theories form a 
complementary network of strands that get to the heart of learner motivation so that each 
stakeholder in the learning process can elevate students to higher achievement (Cook & Artino, 
2016). Drawing on Cool and Artino (2016) and Aguilar (2016), we succinctly summarize the 
contemporary theories about academic motivation in the following sections. 


Expectancy-Value Theory (EVT) posits that the expectation of success and perceived task value 
are two key factors that affect students’ learning behaviors and decision-making process (Eccles 
& Wigfield, 2002). The expectations serve as an actual self-assessment of the likelihood of 


success or failure when a task is attempted (Aguilar, 2016). Eccles and Wigfield (2002) propose 
that task value includes four dimensions: attainment value (e.g., personal importance), the 
intrinsic value (e.g., interest), utility value (usefulness), and cost (e.g., loss of time and stress). 
Expectancy-Value Theory was used to evaluate the design of visualizations within learning 
analytics applications (Aguilar, 2016). 


Attribution theory presents three types of explanations that explain how individuals interpret 
casualty after an event. There are three dimensions in this theory: locus (e.g., internal/external), 
stability (e.g., stable/unstable), and controllability (e.g., controllable/uncontrollable). 
Attribution theory postulates that humans often subconsciously make sense of their success or 
failure via establishing a cause-effect relationship across the three dimensions mentioned above 
based on the events in their lives (Cook & Artino, 2016). 


Social-cognitive theory can be categorized as a theory of learning due to its emphasis on 
humans’ interactions with their environment (Bandura, 1989). This theory postulates that human 
learning results from reciprocal interactions among personal, behavioral, and environmental 
factors. Self-efficacy belief is a vital vehicle to drive the action. Self-efficacy is not the same as 
self-concept or self-esteem, which have broader notions. Instead, it is a domain, task, and 
context-specific notion (Cook & Artino, 2016). For example, an expert might report relatively 
high self-efficacy for performing a job in his or her field of expertise but may have much lower 
self-efficacy for other occupations. 


Goal-orientation theory is also known as Achievement Goal Theory (Barron & Harackiewicz, 
2001). There are four dimensions within AGT: mastery-approach (e.g., learners intrinsically want 
to gain knowledge), mastery-avoidance (e.g., learning is driven by fear of failure and fear of 
looking “bad” compared to others), performance-approach (e.g., eagerly seek opportunities to 
publicly demonstrate competence), and performance-avoidance (e.g., being reluctant to 
undertake tasks out of fear) (Elliot & McGregor, 2001). This theory is often used to distinguish the 
two dichotomous types of mindsets, fixed and growth mindset (Cook & Artino, 2016). Growth 
mindset, or mastery orientation, refers to the belief that ability is malleable, and situations are 
controllable, while fixed mindset, or performance orientation, refers to the belief that ability is 
fixed and situations are less controllable (Cook & Artino, 2016). 


Self-determination theory (Deci & Ryan, 2000) posits that internal autonomy gives rise to human 
action because humans desire to voluntarily act on things that are inherently interesting or 
enjoyable to them (Cook & Artino, 2016). Such ownership or free will is also recognized as 
intrinsic motivation, which refers to acting from inherently enjoyable. Its opposite is named 
amotivation which means people have no intention of doing any activities. Lying in the middle of 
these two types of motivation is extrinsic motivation which refers to acting from the external 
rewards/punishment, contingent self-esteem, guilt, valued goals (Cook & Artino, 2016). SDT 
researchers showed that intrinsic rather than extrinsic motivation is central to learning and to 
creating lifelong learners (Deci & Ryan, 2000; Spinath & Steinmayr, 2012). 


Expectancy Value 


Overview of Expectancy-value Theory (EVT) 
Expectancy-Value Theory (EVT) posits that the expectation of success and perceived task value, 
are two key factors that affect students’ learning behaviors and decision-making processes 


(Eccles & Wigfield, 2002). The expectations serve as a self-assessment, regarding the likelihood 
of success or failure when a task is attempted (Aguilar, 2016). Eccles & Wigfield (2002) propose 
that task value includes four dimensions: attainment value (personal importance), the intrinsic 
value (interest), utility value (usefulness), and cost (loss of time and stress). Expectancy-Value 
Theory was used to evaluate the design of visualizations within learning analytics applications 
(Aguilar, 2016). 


Definition 

Eccles et al. (1994) developed and assessed the expectancy-value model of achievement 
motivation as a framework to explain children’s and adolescents’ performance, persistence, and 
choice of achievement tasks. EVT identifies two constructs that influence the students’ 
expectations (Wigfield, 1994; Wigfield & Eccles, 2000; Eccles & Wigfield, 2002; Wigfield & 
Cambria, 2010). The first construct is the student’s expectations for success or the degree to 
which students believe they will be successful in completing a task. The second construct is the 
individual's task-value beliefs, which is the degree to which students perceive that a task is 
essential and desirable for them to accomplish. A unique characteristic of this construct is its 
subjectivity, as individuals’ values might vary to the same activity. For example, achievement in 
mathematics may be valuable to students pursuing engineering but not to students interested in 
history (Wigfield & Cambria, 2010). 


There are four significant components of task value beliefs within EVT studies (Eccles et al.1983; 
Eccles et al., 2002; Wigfield, 1994; Wigfield & Eccles, 2000; Wigfield & Cambria, 2010). These 
include attainment value or importance, intrinsic or interest value, utility value or usefulness of 
the task, and cost. Attainment value is the importance of performing well to maintain the integrity 
of self-schema. When tasks are associated with high attainment value, such as competence, 
individuals may be motivated to demonstrate quality in their performance to maintain a 
consistent view of themselves. Intrinsic value relates to individuals’ intrinsic motivation in 
accomplishing a task. Such inherent value is subjective and often reflected by a sense of 
enjoyment when the individual performs the task. Utility value speaks to the assessment of how 
the task at hand matches current and future goals. Depending on the relation between the task 
and goals, utility value can be either extrinsic or internalized or both. Cost is the only component 
that captures the negative aspect of a task. For instance, making one choice not only involves 
evaluating the amount of effort needed to engage in the current task but also may deprive the 
person of the benefits of choosing other opportunities, as well as potential harms that may result 
from participating in the task (Eccles et al.1983; Eccles et al., 2002; Wigfield, 1994; Wigfield & 
Eccles, 2000; Wigfield & Cambria, 2010). 


Drawing on the studies in EVT (Eccles & Wigfield, 2002; Wigfield & Eccles, 2000; Wigfield & 
Cambria, 2010), a diagram (See Figure 1) has been constructed that shows the relationship 
between the aspects of EVT. Eccles’ (1983) modern expectancy-value model builds upon 
Atkinson’s (1964) expectancy-value model. One of the substantial developments in Eccles’ 
model is that both the expectancy and value constructs are more concrete, and both can be 
affected by psychological (i.e., task-specific beliefs), social, and cultural factors. According to 
Eccles et al. (1983), both the expectancies and task-values beliefs components determine the 
individual behaviors of performance and task choice. Both parts are directly influenced by 
psychological factors, that is, task-specific beliefs. These beliefs include perceived competence, 
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perceived difficulty of the task, individual goals and self-schema, and the individual’s affective 
memories about past experiences. In addition to the psychological factors, expectancy and 
value can also be directly and indirectly affected by other social and cultural factors, such as 
others’ beliefs about the subject, as well as cultural stereotypes. 
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Figure 1. Eccles and colleagues’ expectancy—value model of achievement motivation 


EVT in Learning Analytics 


Learning Analytics 

There is a long history of Learning Analytics (LA) in education (Knight & Buckingham Shum, 
2017), and anything related to reflecting on learners’ achievements and patterns of behavior 
concerning others can be considered LA (e.g., Statistics). LA has been reintroduced in 
educational technology and has become a highly relevant topic in the education industry. LA 
combines the development of computational and mathematical theories, machine learning, 
social networking, data mining, artificial intelligence, content analysis, and adaptive learning. 
The International Conference on Learning Analytics (2011) composed a definition of LA that 
solidifies its importance in data measurement, collection, analysis, and reporting (Siemens & 
Baker, 2012). This definition emphasizes three pivotal elements: data collection, measurement, 
and action, which endow LA with new dimensions in technology-rich settings. 


Data collection, the fundamental act of LA, enables the collected data to be more personal and 
relevant to individuals with the help of the advanced computational methods such as: 
educational data mining, web analytics, social network analysis, and natural language 
processing (Clow, 2013). For example, students’ time-spent on different learning activities or 
their interaction frequency with peers can be monitored and assessed. LA assesses learning 
outcomes objectively, which shifts the focus of assessment to high-quality formative assessments 
rather than summative characterizations of learning, as occurs with the traditional large-scale 
summative assessment. One unique feature of formative assessment is that the use of assessment 
information is a part of the ongoing learning process and integrally connects with the curriculum 
and instruction that can reflect students’ learning progression (Pellegrino, 2014). Consequently, 
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related stakeholders can harness the power of LA to trace every student’s path of mastery to 
make interventions at an individual level. Action, the unique aspect of the definition, reflects the 
goal of the LA technique which is optimizing learning and the environments in which it occurs. 
Based on these three foundations, LA can lend real-time insights into the decision-making 
process for the related stakeholders by making sense of personal learning information. 


LADs (Visualizations of academic information) 

The personal learning data that LA collects would require tools for students to use to review and 
analyze their own learning history. Dashboard, a tool for monitoring and understanding business 
briefly, was borrowed from the Business discipline (Few, 2006). LA dashboards came into being 
to answer the call for emphasis on the analysis of learning progression to measure what learners 
are experiencing in the online learning environment. Since the Learning Analytics Dashboards 
(LADs) research field is still relatively young, most studies are exploratory and proof-of-concept 
studies (Schwendimann et al., 2017). Different definitions of dashboards exist due to its 
multifaceted nature. For example, researchers have made a distinction between various types of 
dashboards based on a variety of target audiences, including administrator dashboards, 
instructor dashboards, and student dashboards (Schwendimann et al., 2017). 


The application of Expectancy-value Theory in IA and LAD 

Aguilar (2016) examined the application learning analytics through the lens of motivational 
theories (e.g., Expectancy-value Theory, Achievement Goal Theory, Attribution theory, Social 
Cognitive Theory). LADs serve a pivotal role for students to evaluate themselves for making 
data-informed learning decisions. Due to the self-evaluated function of LADs, students’ 
understanding of such data visualizations might influence both expectancies and values to some 
extent. The task-values construct has four components that can function to illustrate the 
effectiveness of data visualization regarding promoting learning outcomes, which could be 
influenced by a learners’ task-value beliefs about their online learning activities. Since 
visualizations are designed to showcase what is necessary, the sense of importance might afford 
“attainment” interpretations. For example, a student who cares about math scores will alter their 
behavior to reach a higher achievement if several examinations reveal low performance. Also, 
the information on the LADs are evolving into the highly self-relevant and personalized 
dashboards, such as customizable LADs (Roberts, Howell & Seaman, 2017). Such meaningful 
visualizations meet students’ needs and afford them intrinsic or interest value for learning. 


In the context of MOOCs, people often volitionally enroll those courses relating to their career 
development. This kind of self-determined learning may be fostered by utility value or 
usefulness of the task components within EVT. Lastly, visualizations that depict comparative 
information might demotivate students with poor academic performance because they would 
feel that the cost of learning is not worth the effort since they are performing below the average 
(Schumacher & Ifenthaler, 2018). 


Blended Learning and EVT 


Our literature search yielded no relevant studies relating EVT and blended learning 
delivery modes. This is an area in need to further research. 


Social Cognitive Theory 

The Social Cognitive Theory (SCT), postulated by Bandura (1986), asserts that people are not 
victims of environmental influences nor do they function as autonomous agents, but rather act as 
partial contributors to their own motivations and actions (Bandura, 1989, 2001). Through a 
complex interchange between self, behavior, and environment, which Bandura (1989) referred 
to as ‘triadic reciprocal causation,” people experience the interplay of these determinants 
through internal processing (past experiences, motivation), external pressure (social context), 
and behavioral responses to their individual personal and environmental influences. In addition 
to the reciprocal determination tenet, SCT accounts for a person's behavioral capacity, vicarious 
or observational learning, reinforcement feedback, individual expectations, and individual self- 
efficacy (La Morte, 2018). 


A central question in the SCT is how people take generative thought and influence it through 
proactive and reflective systems, not just through reactive systems (Bandura, 2001). 
Furthermore, SCT addresses the difference between the physical dimension of thought and its 
purposeful construction and use in functional ways, called acts of human agency (Bandura, 2001). 
Human self-determination is a combination of several human capabilities, namely, intentionality 
(symbolizing), forethought, observation, self-reaction, and self-reflection (Bandura, 1986; 2001; 
Stajkovic & Luthans, 1998). These five aspects of agency play significant roles in human self- 
motivation, which underpins both planning and action through the use of future-directed 
visualization, expectation evaluation through goal setting, competency development, conduct 
regulation, and examination of self-efficacy (Bandura, 2001; Bratman, 1999). Perceived self- 
efficacy is the linchpin in SCT due to its influence on personal adaptation and its impact on the 
other elements in the SCT (Bandura, 2001). 


Bandura found that personal development is a central characteristic of human activity and part of 
being human is the concept of personal empowerment to, at least, aim toward actions that 
produce desired outcomes or prevent undesirable outcomes. Furthermore, individuals take this 
human agency into the realm of society through a collective agency. Bandura asserts that 
collective agency translates into a perceived collective efficacy that elevates groups to imagine 
achievement in a broader, societal sense. This results in a greater motivational ethic to 
commitments, greater resilience to adversity, enhances performance, and impacts educational 
self-development (Bandura, 2002). 


The technological age holds the promise of helping learners of all ages and backgrounds to 
engage in online content for educational purposes. However, the age-old specter of learner 
motivation attenuates the hopefulness of what humanity and technology can accomplish in 
teaching and learning (Bandura, 2002). Students must develop cognitive skills, along with 


adeptness at self-regulation, as strong self-regulators flourish with increased motivation, greater 
degrees of self-efficacy, and improved academic achievement (Zimmerman, 1990). In addition to 
self-regulatory skills, learners must bring self-management prowess into play and continue to 
pursue goals in the face of obstacles (Bandura, 2002). 


The Social Cognitive Theory in Online Learning 


Reciprocal Determinism 

The reciprocal determinism construct holds some credence in the development and functioning 
of online courses. Bautista (2013) found that leveraging previous online interactions with which 
students were familiar, such as online chats and virtual worlds, showed a high correlation to 
students’ receptivity, participation, and success in scaffolding online discussions in Physics 
learning. Additionally, a very high positive correlation was found between a priori experience 
and student participation, formative, and summative assessments (Bautista, 2013). Reciprocal 
determinism is measurable and positively associated with students’ abilities to mimic the 
complexity of discourse in an instructor’s posts, in threaded discussions but not in terms of the 
word count (Ryan-Rojas, Douglass, & Ryan, 2012). 


Vicarious (observational) Learning 

Vicarious learning refers to the process of acquiring knowledge from non-involved observation 
(Bandura, 1986; Gholson & Craig, 2006; Mayer & Chandler, 2001). Vicarious learning can occur 
frequently in online learning scenarios that place students as observers rather than active 
participants in questioning or discussion (Gholson & Craig, 2006). Much research has been 
directed toward discerning what manipulations can influence online learning (Craig, Sullins, 
Witherspoon, & Gholson, 2006; McNamara, Levinstein, & Boonthum, 2004). 


One mechanism for providing a rich observational environment is using animated agents (Craig, 
Gholson, & Driscoll, 2002). Because they can promote a sense of interest, and emotional 
overtones, animated agents can provide some features of human-human interactions that 
learners find engaging and motivational (Andre, Rist, & Muller, 1999; Craig, et al. 2002). These 
agents are not detrimental to learning and have been shown to improve skill learning during 
transfer tests (Craig, Gholson, Ventura, Graesser, & the Tutoring Research Group, 2000; Twyford 
& Craig, 2017). 


Another tool for use in vicarious online learning is the use of deep level questioning as a dialog 
feature compared to a dialog of lesser depth (Craig, Gholson, Brittingham, Williams, & Shubeck, 
2012). Deep reasoning questions result in higher post-test scores for learners compared to 
controls (Sullins, Craig, & Graesser, 2010). 


Still another tool for affecting vicarious learning is modeling, which has always been the 
hallmark method of transmitting values, attitudes, thought patterns, and appropriate behavior 
(Bandura, 1986). Models activate and support appropriate behavior (Bandura, 1986; Twyford & 
Craig, 2017) and have been shown to be instrumental in improving goal setting and knowledge 
retention in multimedia environments (Twyford and Craig, 2017). Notably, models who initially 
demonstrate undesirable behaviors but transition into improved performance and confidence by 
employing coping strategies are impactful on improving posttest scores and improving self- 
efficacy measures (Braaksma, Rijlaarsdam, & van den Bergh, 2002). Modeling techniques can 
serve to demonstrate collaboration and questioning skills that can be transferred to any learning 
setting (Craig & Brittingham, 2013). 


The advantages of tutoring to aid learning are well established (Chi, Roy, & Hausman, 2008; 
Twyford & Craig, 2017; VanLehn, 2011). Tutoring allows one-to-one interactivity and improved 
instructor engagement to deepen instructional interactions (Chi et al., 2008). Combining the 


power of tutoring with social and cognitive advantages of modeling and collaboration have 
shown promise in student learning (Chi et al., 2008). Van Lehn (2011) demonstrated that 
systems-based tutoring that intelligently guides learners is as beneficial to learning as human 
tutors. Overhearing tutoring dialogue that incorporates deep learning questions resulted in 
improvement in both the amount of and relevance of student-generated content (Craig et al., 
2000). 


Reinforcement Feedback 


Feedback falls into three basic categories: the knowledge of results, the knowledge of the 
correct response, and elaborated feedback (Wang & Wu, 2008). In web-based learning 
environments, student elaborated feedback was related to higher self-efficacy and students who 
received elaborated feedback had a promotion in self-efficacy due to the better quality of the 
feedback given (Wang & Wu, 2008). 


Self-efficacy 


Students’ efficacy beliefs originate from self-persuasion which is informed through the cognitive 
processing of efficacy information (Bandura, 1989). Self-efficacy is distinguished from related 
concepts, such as self-esteem, in that it is largely performance-derived and has predictive 
discriminant validity for academic outcome measures (Bradley, Browne, & Kelley, Hodges, 2008; 
Lin & Overbaugh, 2009; Zimmerman, 2000). Self-efficacy originates from the combination of self- 
evaluations on performance mastery, vicarious experiences of observation and comparison 
(modeling), verbal persuasion and other social influences that indicate proficiency, and 
physiological states that tell us about our own ability (Bandura, 1986; 1989; Freudenberg, 
Cameron, & Brimble, 2011; Zimmerman, 2000). 


Based on this “cognitive weighing process”, a person’s assessment of whether they can do 
something changes from situation to situation (Hodges, 2008; Zimmerman, 2000). Personal self- 


efficacy beliefs determine how much effort learners will put forth in online classes and how long 
they will persist when obstacles arise; therefore, educators should extend the focus of learning 
to include students’ perceptions of ability and actual demonstrable ability (Hodges, 2008; Lin et 
al., 2009). Zimmerman (2000) stated that self-efficacy is sensitive to subtle changes in the context 
of performance, that it is interactive with self-regulated learning schemas, and mediates 
academic achievement. 


In summarizing Bandura’s work, Martin (2004), stated that self-efficacy is the most important 
aspect of self-regulation and self-regulation operates under the umbrella of self-determination. 
Self-efficacy encompasses both general beliefs an individual holds about their personal 
capabilities and task-specific beliefs (Bandura, 1989). Since self-efficacy beliefs are task- 
specific, changing learning modes from face-to-face encounters to online learning modes may 
affect a students’ self-efficacy (Hodges, 2008). 


Self-efficacy cannot be viewed without remembering the disposition of human agency in using 
vicarious means to evaluate oneself (Hodges, 2008). In experiments in which students received 
feedback on their own performance after viewing a modeled performance, improvements in 
persistence and self-efficacy were demonstrated (Hodges, 2008; Schunk, 1987). In research that 


evaluated the role of peer-modeling to self-efficacy beliefs, self-efficacy affected a student’s 
choice of activities and instructional interventions (Schunk, 1987; Zimmerman, 2000). 


One such choice activity that is relevant to online instruction is the initial choice students must 
make regarding asynchronous versus synchronous online instruction. Lin et al. (2009) found that, 
in hybrid courses, two-thirds of students preferred asynchronous modes of interaction versus 
synchronous modes for conducting computer-mediated discussions. This preference was 
evident across genders. However, participants in the synchronous mode reported more 
efficacious perceptions than those in the asynchronous mode of learning in the subdomain of 
participation web activities. No differences in self-efficacy between learning modes were 
revealed in the subdomains of information literacy, learning theory, problem-based learning, 
cooperative learning, or general online communication. In general, females were likely to 
perceive themselves as more efficacious, but significance was only reached in the subdomain of 
learning related learning theory (Lin & Overbaugh, 2009). 


Recently, Du and colleagues (2019) found three student variables that were related to self- 
efficacy in online collaborative learning. These variables, all of which exist at the student level 
are: 1) willingness to navigate and persist the challenge of groupwork, 2) experiencing a 
trusting relationship with other members of the group, and 3) sensing an influence from the 
course leadership. Their recommendations for establishing course leadership include designing 
high-quality group projects that are purposeful, meaningful, challenging and engaging (Du et 
al., 2019). Aubert and Kelsey (2003) found that trust was independent of successful group 
performance but that information symmetry (for example, ‘quality’ work had different definitions 
between participants), and good communication distinguished high-performing groups from 
low-performing groups. Teams that were low-performing had different perceived goals, 
communication difficulties, and lack of attending to known group-interaction problems (Aubert & 
Kelsey, 2003). De Dreu and Weingart (2003) found that relationship- and task-conflict had strong 
and negative correlations with team performance and satisfaction. Conflict had higher negative 
correlations in situations where the task was more complicated, such as during decision making 
and project work versus production work. 


MO00Cs/At-Scale and Social Cognitive Theory 


Several search attempts were made to find research data directly linking Social Cognitive 
Theory with MOOCs, and Social Cognitive Theory with Learning At-Scale. Little to no information 
was found showing that research has been done on the connection between these two topics. It is 
highly plausible that modeling with video which has been shown to be highly effective for 
learning in computerized environments (Chi et al., 2008; Craig et al., 2006; Craig et al., 2009; 
Gholson & Craig, 2006) would transfer to MOOCs and learning at scale. As it stands, this report 
would indicate that such research is critically needed for the betterment in understanding 
MOOCs/At-Scale and the direction of future learning. 


Blended Learning and Social Cognitive Theory 
Self-efficacy has been shown to improve using blended learning experiences in teacher 
professional development courses (Abello, 2018). This effect was a consequence of improved 
modeling and collaborative skills, mastering blended learning skills through positive feedback, 
improved communication, and self-regulation (Abello, 2018). Al Fadda (2019) found that a 


positive and significant correlation existed between self-efficacy and course grades and verbal 
ability and course grades; however, goal orientation, internet self-efficacy, time spent at 
studying, study environment, or help seeking behaviors were not associated with course 
performance. This supports earlier work that found a significant and positive relationship 
between success in online courses and self-efficacy. In blended learning environments, help 
seeking behaviors, internet self-efficacy, time studying, and the study environment may not be 
factors since there is regular interaction between students and the instructor (Al Fadda, 2019). 


Self-Determination Theory 

Researchers strive to figure out what motivates online students. Self-Determination Theory (SDT) 
is a prominent and well-established theory about motivation to learn established by Deci & Ryan 
(1980). SDT is considered a macro theory of human motivation because it addresses such basic 
issues as personality development, self-regulation, universal psychological needs, life goals and 
aspirations, energy and vitality, nonconscious processes, the relations of culture to motivation, 
and the impact of social environments on motivation, affect, behavior, and wellbeing (Deci & 
Ryan, 1980; 2008). 


There are two major concepts within SDT that researchers have explored, which are the 
concepts of goal contexts (i.e., autonomy-supportive environments versus controlling social 
environments) and the concepts of goal contents (i.e., intrinsic versus extrinsic personal goals) 
(Deci & Ryan, 2008; Ryan & Deci, 2000, 2006; Vansteenkiste, Lens & Deci, 2006). The former 
focuses on what social context is related to learning; the latter focuses on what content of the 
goals people pursue. 


The concept of goal contexts includes autonomy-supportive environments versus controlling 
social environments. Vansteenkiste and colleagues (2006) proposed a definition for autonomy- 
supportive environments based on Deci and colleagues’ (1980) description of this concept. 
Autonomy-supportive environments were described as instructors endorsing a learner’s 
learning choice through timely positive feedback, providing meaningful rationale if the choice is 
constrained, and refraining from the use of pressures and external rewards to motivate behavior 
(Vansteenkiste et al., 2006). 


Contrastingly, Vansteenkiste and colleagues (2006) concluded that the second type of goal 
context —controlling social environments—tends to overly pressurize individuals into engaging 
in the learning, such as setting deadlines or using controlling language (e.g., “have to,” 
“should” and ‘‘ought”). They emphasized the unique role of autonomous motivation in SDT. 
Autonomous motivation is where choices originate with the learners (Deci & Ryan, 2008; Ryan & 
Deci, 2000, 2006; Vansteenkiste, Lens & Deci, 2006). Vansteenkiste and colleagues further 
proposed a motivational mediation model (Figure 1) indicating a positive association between 
the autonomy-supportive environments and learning outcomes (e.g., deep-level learning, 
greater achievement, and higher persistence at learning activities). 


Secondly, the concept of goal contents includes intrinsic versus extrinsic goal framing (Deci & 
Ryan, 2008; Ryan & Deci, 2000, 2006; Vansteenkiste, Lens & Deci, 2006). Vansteenkiste et al. 
(2006) and Ryan and Deci (2000, 2006) contrast intrinsic goal framing (e.g., personal growth, 

meaningful relationships with others, becoming more healthy and fit, or contributing to the 
community) from extrinsic goal framing (e.g., fame, financial success, and physical appearance) 
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with respect to student engagement in the learning activity and student academic performance. 
Many SDT studies indicated that intrinsic goal framing in an autonomy-supportive conditions led 
to higher autonomous motivation, better test performance, and greater persistence than in the 
controlling context (Vansteenkiste et al., 2006; Ryan and Deci, 2000, 2006; Deci and Ryan, 2008). 
Likewise, SDT studies (Vansteenkiste et al., 2006; Ryan and Deci, 2000, 2006; Deci and Ryan, 
2008) maintain another motivational mediation model regarding goal contexts, as demonstrated 
in Figure 1. It is proposed that autonomous motivation also mediates the predictive effect of goal 
contents on learning outcomes. 


Goal Contexts: 
Autonomy-supportive | 
Environments versus } 
Controlling Social 
Environments 


Autonomy 


Motivation Learning Outcomes 


Goal Contents: 
Intrinsically and 
Extrinsically oriented 


Figure 1. SDT Motivational Mediation Model 


Self-Determination Theory in Distance Learning Practice 


Andragogy and Heutagogy 

Andragogy is associated with SDT (Cercone, 2008; Beaven, Hauck, Comas-Quinn, Lewis, & de 
los Arcos, 2014). There is a close connection between SDT and andragogy that may contribute to 
design elements in the online learning environment. For instance, Cercone (2008) presents five 
assumptions of andragogy that will effectively motivate online adult learners to be more 
engaged and self-directed. These assumptions primarily describe what adult learners might be 
like and what kind of instructions teachers should provide to help them become self-directed 
learners. Based on Canning (2010) and Blaschke (2012), there is a pyramid-type Pedagogy-— 
Andragogy—Heutagogy Continuum in which learners’ maturity and autonomy increase and 
instructors’ control and course structuring decrease from pedagogy to andragogy to heutagogy. 
Beaven and colleagues (2014) pointed out a relationship between heutagogy and SDT because 
these two concepts are both intrinsic to learner autonomy and the experience of choice. 
Specifically, andragogy focuses on supporting self-directed learning, while heutagogy focuses 
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on promoting self-determinate learning by endorsing one's actions with a full sense of choice 
and volition (Blaschke, 2012). 


SDT in the online learning environment 


Empirical studies have supported the mediating effect between the core concepts within SDT in 
the online learning environment (Chen & Jang, 2010; Kennan, Bigatel, Stockdale & Hoewe, 2018). 
The three core constructs in SDT, namely, autonomy, relatedness, and competency, have 
potential to address learning problems such as poor engagement, low academic achievement, 
and student attrition in online settings (Chen & Jang, 2010). For example, Chen and Jang (2010) 
have shown the intricate dynamics among contextual support, need satisfaction, autonomous 
motivation, and learning outcomes through SDT full model tests. In their study, Chen, and Jang 
(2010) found that contextual support (e.g., encouraging autonomy and competence support) 
combines with need satisfaction (e.g., perceived autonomy, perceived relatedness, and 
perceived competence) to produce an impact on students’ learning outcomes. However, 
autonomous motivation failed to directly predict any of the learning outcomes in their study 
(Chen & Jang, 2010). 


Kennan et al. (2018) found evidence of the relationship between self-determination and adult 
learners. Such relationships include evidence of the self-determination continuum posited by 
Deci et al. (2000) that ranges from a motivational (e.g., low perceived self-relevance) to intrinsic 
motivation (e.g., high perceived self-relevance), as well as the assumptions that underlie 
andragogy (e.g., meets learners’ needs, career-oriented education, etc.). Kennan et al. (2018) 
raised the question of whether online instructors need to consider more practical considerations 
in online pedagogy and instructional design, such as the impact of age and class standing for 
online students in future SDT-based studies. (Kennan et al., 2018) 


Also, Seiver and Troja‘s research findings (2014) suggest that it is not only essential to satisfy the 
needs of the majority of students in the courses by providing task-oriented and well-structured 


courses, but also to include off-topic discussions, collaborative work groups, and personalized 
communications from instructors in online courses in order to increase retention and student 
satisfaction among those students who have the highest needs for affiliation with online classes. 
However, the need for autonomy was not associated with satisfaction and success in online 
learning (Seiver & Troja, 2014). 


These conclusions seem contradictory with what SDT has proposed. However, it also confirmed 
that the complex nature of the application of SDT in the online learning environment, and a ‘one 
size fits all’ approach in an online context does not work. The findings in Hartnett, George and 
Dron’s study (2011) shows that autonomous motivation with insufficient scaffolding is detrimental 
for online learners because they may feel lost and unsupported. The incompatibility of the 
asynchronous communication medium and the frequent, ongoing, and collaborative problem- 
based learning activities is another challenge of online learning design. These findings shed 
light on rethinking the application of SDT in the online learning environment because of the 
dichotomous conceptualization of autonomous and controlling motivation within SDT. These two 
contradictory concepts could hardly explain student perceptions about online learning due to 
the diversity of its users and the various confounding variables in the online setting. Applying 
SDT directly into practice, causes many challenges. Practice (Hartnett, George and Dron, 2011). 
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M00Cs 

MOOCs are often the setting in applying SDT in practice (Beaven et al., 2014; Loizzo, Ertmer, 
Watson & Watson, 2017; Martin, Kelly & Terry, 2018). Due to the nature of the MOOCs are 
heralded as being free-choice and autonomous learning opportunities, SDT as a prominent 
academic theory could be used to explain the motivational aspects of the design and use of 
MOOCs. Beaven and colleagues (2014) found that the effectiveness of SDT might vary for 
different types of MOOCs (e.g., task-based, content-based, and network-based). They also 
conclude that the participants in task-based MOOCs assume more self-determination and a 
higher degree of participatory literacy than those with a content-based focus (Beaven et al., 
2014). 


To gain a more conceptual understanding of the adult learners’ MOOC experience, Loizzo et al. 
(2017) propose a conceptual framework of adult learner MOOC motivations and goals. Using 
virtual ethnographic methods to examine the online learning experiences of 12 adult learners, 
they identified three prevalent motivators that are influential in learners’ motivation to enroll and 
participate in the Human Trafficking MOOC. These motivators were professional development or 
volunteerism, enjoyment of MOOCs, and flexibility of MOOCs. In line with SDT, the sense of 
choice and volition is paramount to the online learners. In a MOOC setting, learners can set their 
own learning goals and making their own decision about the degree of participation and the 
level of engagement. However, current MOOCs often do not provide learners with the 
opportunity to assess their own progress in relation to their personal goals (Loizzo et al., 2017). 


Blended Learning and Self-Regulation and Self-Determination 


Varthis and Anderson (2016) found that blended learning environments increased learner 
motivation, improvement in learning skills, active learning, perceptions of learning quality, and 
student self-regulation. Van Laer and Elen (2017) found that there were seven attributes of 
blended learning environments that could promote self-regulation: 1) authentic tasks, 2) tailored 
learning experiences, 3) learner control of pace, content, sequence, and learning activities, 4) 
scaffolding that helps students bridge their current zone of proximal development, 5) learner 
collaboration with the instructor and other students, 6) using cues to signal learners to reflect on 
critical content, and 7) learner calibration processes that allow learners to evaluate their own 
performance. Van Laer and Elen (2017) suggested that blended learning may prove more 
challenging for lower self-regulated students than for highly self-regulated learners. Silva, 
Zambom, Rodrigues, Ramos, and de Souza (2018) found that providing students in a blended 
learning environment with learning analytics feedback at frequent intervals improved student 
self-regulation. In an English language blended learning class, self-evaluation was found to be 
the single most important factor in a student’s self-efficacy and self-efficacy was predicted by 
goal setting (Su, Zheng, Liang, and Tsai, 2018). Further, Su et al. (2018) found that students who 
were able to exercise self-regulation in their learning environment structuring increased the 
learner’s self-efficacy in reading and writing. 


Shea and Bidjerano (2010) found that self-efficacy and teacher presence were stronger ina 
blended learning environment. Learner self-regulation and self-efficacy (learner presence) must 
be supported in blended learning by fostering metacognitive, emotional, and behavioral 
aspects of students by helping them become active learners. Weaker students can benefit most 


from teacher support through teaching self-awareness which is fostered by self-reflection 
activities (Shea and Bidjerano, 2010). 


The Attribution Theory 


In the case of attribution theory (AT), motivation is analyzable at the level of variability in 
reactions of one person to similar events (Cook & Artino, 2016). For example, after an incident 
happens, if the expected outcome occurs, a person may directly experience emotions like 
frustration (if the predicted outcome was negative, as expected) or happiness (if the predicted 
outcome was positive, as expected). However, when the result is unexpected, negative, or 
perceived by the person as important, it awakens a desire to understand the cause of the event. 
A person seeking to settle upon a reason for an outcome perceives the incident in specific 
environmental and personal conditions and will make an attribution that fits the perceived 
circumstances, such as luck, mood, another person, ability, effort, and so on. 


Attribution theory presents three types of attributions that explain how individuals interpret 
casualty after an event (Weiner, 1979). These three attribution types are empirically 
demonstrated pathways along which interpretation of the events (attribution) can occur, namely, 
locus (e.g., internal/external; whose fault was the outcome?), stability (e.g., stable/unstable; is 
the outcome changeable for future events?), and controllability (e.g., 
controllable/uncontrollable; is the outcome modifiable by a person?) (Cook & Artino, 2016; 
Weiner, 1985, 2005, 2010). Attribution theory postulates that humans often subconsciously make 
sense of their success or failure via establishing a cause-effect relationship across these three 
dimensions based on the events in their lives. Further, attributions alone do not provide 
motivation, but instead attributions are psychologically reframed by a person into actionable 
responses (Cook & Artino, 2016). For example, if a person is late to work one day (unexpected 
outcome) because of heavy traffic (uncontrollable) due to emergency road construction 
(external locus) for a damaged guardrail (unstable) they may not feel the need to modify their 
morning routine (reframed into an actionable response). However, changing the conditions of 
the example can change the locus, stability, and controllability perceptions of the person, which 
may result in a change of action. 


The Significance of Attribution Theory to Learning 


Learners frequently search for causes of learning performance and behavior. Attribution Theory 


(Weiner, 1979; 1985) focuses on explaining ‘‘why” learners have different reactions to a given 
learning experience. AT posits two motivation perspectives: the intrapersonal versus 
interpersonal perspective. As its name indicates, the intrapersonal theory of motivation 
describes self-directed emotions, such as pride or self-esteem. The interpersonal theory of 
motivation pertains to other-directed emotions, such as sympathy and anger (Weiner, 2005; 
Graham & Williams, 2009). 


There are links between the outcome attribution, the emotional feeling, and the chosen action 
which infer that thinking gives rise to feelings that guide actions (Weiner, 1979; 1985; 2007; 2010; 
Hareli & Weiner, 2002). When put into the context of the AT, learner's perspective of locus 
(whether a cause is internal or external to the individual), stability (whether a cause is constant 
or varying over time), and controllability (whether a cause is subject to deliberate alteration) 
affect the student’s decision to alter their learning approach. Examples include perceptions such 
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as, student ability and effort are internal causes of success, whereas luck and help from others 
are external causes. Aptitude is perceived as constant, whereas luck is viewed as unstable or 
temporary. Luck and ability are unalterable, whereas effort is controllable by the individual. 
Each event is filtered through the student’s personal grid and given an attribution. Attribution 
theorists believe that all causes, theoretically, fall into one of the eight cells of a dimensional 
matrix defined by locus x stability x controllability (Graham & Williams, 2009). 


When motivation is initiated by achievement outcomes (e.g., achievement failure), then the 
causal search looks at others (teachers or peers) as responsible. For instance, in a school setting, 
teachers usually express sympathy to a low-performing student because, in this case, failure is 
attributed to a lack of aptitude, which is an unintentional and noncontrollable cause. However, if 
teachers think one’s failure is due to an intentional lack of effort, it might elicit anger so that 
punishment or reprimand will follow (Graham & Williams, 2009; Weiner, 2005). 


According to Weiner’s work (2005), intrapersonal and interpersonal perspectives are 
interrelated. Sometimes, what appears positive in the interpersonal context (e.g., expression of 
an antisocial emotion) has negative consequences for personal motivation. For instance, help 
offered by a teacher may trigger negative thoughts about learning ability in a student (e.g., 1am 
not smart, and I always need help). This situation will demotivate students in their pursuit of 
knowledge. Therefore, understanding the interrelations of these two perspectives in educational 
practice is crucial because it accounts for the discrepancy between students’ own experiences of 
success and failure and others’ impressions of them (Weiner, 2005). 


The Application of Attribution Theory - Attribution Retraining (AR) Treatments 


Attributional Retraining (AR) is an attribution-based intervention for young adults in competitive 
achievement settings which includes tactics such as encouraging students to adopt controllable 
and unstable explanations for academic failures, such as a lack of effort or a weak study strategy 
(Hall, Hladkyj, Perry & Ruthig, 2004; Hall et al., 2007). Some empirical findings show that AR 
improves students’ performances, as evidenced by higher course grades and GPAs (Grade 
Point Average). Since better achievement usually leads to a positive emotion, AR is indirectly 
related to positive emotions (Perry, Stupnisky, Hall, Chipperfield & Weiner, 2010; Ruthig, Perry, 
Hall & Hladkyj, 2004; Hamm, Perry, Chipperfield, Murayama & Weiner, 2017). Specifically, 
Hamm and colleagues (2017) conducted a study to examine attributional retraining (AR) and 
stress-reduction (SR) treatments in an online learning environment to first-year college students 
who differed in cognitive elaboration skills (low versus high). They found that AR treatment hada 
significant effect on low-elaboration students by reducing the maladaptive attributions and 
increasing the positive emotions about academic performance. This finding is in line with Hall 
and colleagues’ (2007) findings. Hall et al. (2007) found that writing-based AR intervention 
improved academic development for both low- and high-elaborating students. A significant 
improvement in course performance for low-elaborating students was detected (Hall et al., 
2007). These findings are relevant in future exploration of AR in the online learning context to 
improve online student retention. The attributional model of motivation is a complex 
interrelationship between thinking, feeling, and acting, which is highly subjective. However, the 
online learning environment presents objective information, such as time-spent, grades, or 
interaction frequency. This information appears on the students’ dashboards. Students have 


accurate data to improve their subjective experiences employing AR interventions to increase 
achievement and persistence. 


There is ample literature on the attribution-emotion linkage (Weiner, 1985; 2007; 2010; Hareli & 
Weiner, 2002). For example, Maymon and colleagues (2018) conducted two empirical studies to 
examine the relationship between causal attributions and emotions concerning academic 
computing difficulties in post-secondary education. These researchers found mixed effects on 
students’ feelings that were elicited by internal and personally controllable attributions for 
computing challenges in two types of learning contexts (traditional and online). Specifically , 
controllable attributions for academic failure were beneficial for student learning, persistence, 
and achievement in a traditional learning setting. However, such attributions were detrimental to 
online students (Maymon et al., 2018). These findings demonstrate that context matters. When 
applied to online learning, stakeholders should remain mindful that the benefit of a personally 
controllable attribution in one context can lead to a very different emotional experience in 
another setting. 


Attribution Theory and MO0Cs/At Scale Learning 


Several search attempts were made to find research data directly linking Attribution Theory 
with MOOCs and Attribution Theory with At Scale learning. Little to no information was found 
showing that little research has been done on the connection between these two topics. As it 
stands, this report would indicate that such research is critically needed for better 
understanding MOOCs and the direction of future learning. 


Blended Learning and Attribution Theory 

Research has shown that students can associate perceptions of their learning experience with 
positive feelings about their learning outcomes (Ellis & Han, 2018). Ellis and Han (2018) found 
that sometimes students avoid blended learning courses because they value working face-to- 
face with others rather than do solitary online work. They found that students may perceive that 
the online portions of the course are unrelated or unintegrated to the course. Additionally, 
students perceived that the use of online materials would increase their workload or that online 
contributions by other students were undervalued, so blended courses were avoided. Students 
who disliked blended learning tended to prefer face-to-face interactions, perceive that the face- 
to-face mode would be less work, and performed at a lower academic level. Students who held 
negative interpretations of the integration of the online course components and who esteemed 
contributions from other students less favorably also performed at a lower academic level (Ellis 
& Han, 2018). 


Mosalanejad, Alipor, and Zanid (2010) found that the mean academic achievement score of 
blended learning participants exceeded that of traditionally taught students. Using the 
Attribution Measurement Test, which measures an individual’s attributions about positive and 
negative consequences Mosalanejad et al. (2010) found that students in unfavorable conditions 
were more likely to attribute this position to internal, unstable, and local causes and to attribute 
favorable conditions to more external, unstable, and local factors (Mosalanejad, Alipor, & Zanid, 
2010). 


Goal Orientation 

Goal-orientation theory is also known as Achievement Goal Theory (AGT) (Barron & 
Harackiewicz, 2001). There are four dimensions within AGT, including mastery-approach (e.g., 
learners intrinsically want to gain knowledge), mastery-avoidance (e.g., learning is driven by 
fear of failure and fear of looking “bad” compared to others), performance-approach (e.g., 
learners eagerly seek opportunities to publicly demonstrate competence), and performance- 
avoidance (e.g., learners are reluctant to undertake tasks out of fear) (Elliot & McGregor, 2001; 
Irvine, 2018). AGT is often used to distinguish between two dichotomous mindsets — fixed and 
growth mindset (Cook & Artino, 2016). Specifically, mastery orientation refers to a growth 
mindset in which ability is malleable and situations are controllable, while performance 
orientation refers to a fixed learning mindset in which ability is fixed and situations are less 
controllable (Cook & Artino, 2016). 


A Brief Conceptual History of the Achievement Goal Construct 

During the last 40 years, scholars like Carol Ames, Carol Dweck, Marty Maehr, and John Nicholls 
continuously revised the achievement goal construct (Elliot, 2005; Elliot, Murayam, & Pecrun, 
2011). The initial achievement goal model, called the dichotomous achievement goal model, 
posited two achievement components designated as achievement goals (e.g., mastery goals 
versus performance goals) and approach goals (Ames, 1992; Elliot et al., 2011). These two 
components later formed a 2x2 achievement goal model by including a distinction between 
approach and avoidance motivations (see Figure 1) (Elliot, 1999; Elliot & McGregor, 2001). This 
resulted in four achievement goal constructs, namely, mastery-approach, mastery-avoidance, 
performance-approach, and performance-avoidance. Notably, there are two fundamental 
dimensions of achievement goals which are competence definition and competence valence. 


Competence is the conceptual core of the achievement goal construct because it represents the 
reasons and aims that people use to evaluate their achievement (Elliot & McGregor, 2001; Elliot, 
2005; Elliot et al., 2011). Competence definition refers to the standard and the referent that 
learners use to determine if they are performing well or poorly. Competence valence refers to 
two behavioral tendencies that are moderated by two temperaments, namely, positive 
temperament (e.g., success and competence) and negative temperament (e.g., failure and 
incompetence). 


Competence Definition 
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Figure 1. Elliot and McGregor’s (2001)2x2 Achievement Goal Model 


Rooted in the 2x2 achievement goal model, Elliot and McGregor (2011) later expanded the 
competence definition into three types of goals which include task-based goals, self-based 
goals, and interpersonal-based goals. These adjustments to the competence definition resulted 
in an achievement goal model framed in a 3x2 matrix with six goal constructs (see Figure 2). 
Specifically, task-based goals use task requirements as the evaluative referent for measuring the 
degree to which one has accomplished the task. Self-based goals use the personal learning 
trajectory of the student as the evaluative referent. Students’ self-based goals help them 
calculate future performance considering past performance for goal setting. Interpersonal- 
based goals use a social-comparison as the evaluative referent, such as how one is doing 
compared to others. 


As shown in Figure 2, a task-approach goal focuses on the attainment of a task-based 
competence (e.g., get the right answers) while a task-avoidance goal focuses on the avoidance 
of a task-based incompetence (e.g., avoid getting the wrong answers). Further a self-approach 
goal targets the attainment of a self-based competence (e.g., get a higher score than before), 
while a self-avoidance goal targets the avoidance of self-based incompetence (e.g., avoid 
getting a lower score than before), Finally, an interpersonal-approach goal centers efforts on the 
attainment of an interpersonal-based competence (e.g., getting a higher score than peers), while 
an interpersonal-avoidance goal centers on the avoidance of an interpersonal-based 
incompetence (e.g., avoid getting a lower score than peers). 


Elliot and McGregor (2011) conducted two empirical studies to support the 3x2 achievement 
goal model. The results from both studies provide evidence to the antecedents and 
consequences of the achievement goal construct. Specifically, the approach and avoidance 
temperaments were predictive of the approach and avoidance goals. These findings replicated 
prior work in which a valence match existed, such that approach-based goals emerged from 
approach temperaments and avoidance-based goals emerged from avoidance temperaments. 
Moreover, task-approach goals positively predicted intrinsic motivation, learning efficacy, and 
content absorption in class, whereas self-approach goals were related to none of these variables. 
Also, these results suggest that the process of mentally contrasting a future possibility with a 
present reality is exceptionally impactful in setting self-approach goals. Finally, interpersonal- 
approach goals facilitated performance, whereas interpersonal-avoidance goals were 
detrimental for performance. They offered guidance in applying this model, which includes 
emphasizing to students the need to discourage the pursuit of interpersonal-avoidance goals and 


recommending the benefits of promoting task-approach over self-approach goals (Elliot & 
McGregor, 2011). 
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Figure 2. Elliot & McGregor’s 3x2 Achievement Goal Model 


Achievement Goal Theory in Distance Learning Practice 

As is the case with Attribution Theory, there is limited literature that characterizes the 
application of the AGT in an online learning context. Aguilar (2016) used a 2x2 achievement goal 
model to frame his studies in the realm of learning analytics, in which he proposed that AGT- 
inspired constructs are also applicable in the digital environment. Utilizing such constructs, 
online instructors can effectively intervene in student learning in terms of information 
visualization for increasing student academic motivation. Aguilar’s (2016) work expands the 
application of AGT to the digital realm in other ways, such as the development of the Motivated 
Information Seeking Questionnaire (MISQ) to measure how students “make-sense” of or 
interpret graphical representations of their achievement data. For example, having students take 
MISQ prior to designing the dashboard will ensure that their general motivational orientation 
towards graphic representations is accurately measured so that students’ dashboards only 
encompass highly self-relevant and meaningful graphic representations. Utilized correctly, the 
MISQ could provide designers with principles in the design of learning dashboards (Aguilar, 
2016). 


Other research points to practical uses of the EVT. Seldow (2010) concluded that accessing the 
online grade book has positively impacted students’ levels of mastery goal orientation 
compared to using the traditional paper-based grade book. However, the performance- 
approach and performance-avoidance goal orientation were not significantly different between 
the online grade book and traditional paper-based grade book groups. 


Crippen and colleagues (2009) conducted a study to examine the roles of goal orientation in 
learning from worked examples (e.g., an instructional technique of pairing a worked example 
with practice problems) to influence academic achievement. Their results indicated that a 
mastery-approach orientation strongly predicted achievement and a mastery-avoidance 
orientation was negatively related to achievement. With respect to the use of worked examples 
as a learning strategy, mastery-approach students may not perceive the worked examples to be 
as useful as the performance-oriented students because the worked example is considered to be 
too easy by the mastery group. Worked examples can be difficult to develop at the necessary 
depth of understanding for mastery-approach students (Crippen, Biesinger, Muis, & Orgill, 
2009). 


Blended Learning and Goal Orientation 

Lewis (2018) found that students in blended chemistry classrooms showed higher task 
orientation than students in traditional classrooms. Lyons, Limniou, Schermbrucker, Hands, and 
Downes (2017) found students with mastery goal orientation and agreeableness had a significant 
and positive association with a preference for flipped classroom arrangements. Alvarez and 
Cuesta (2011) suggested that helping students set attainable goals contributes to self-efficacy in 
blended learning environments. Such assistance can take the form of articulating explicit 
learning objectives, providing students with estimates of time for task completion for each 
activity, and providing specific goals for each activity (Alvarez & Cuesta, 2011). Leung (2012) 
found that students learn better when they are provided with specific learning outcomes and that 
effectiveness in learning is situational in that environmental conditions can alter a student’s self- 
efficacy. Leung (2012) stated that students experience more self-efficacy at the semester’s end 
when due dates approach and that during this time, students spend less time on goal-setting and 
class activities and spend more time on writing up their assignments that are nearing due dates. 
Leung also found that freshman students experienced higher academic self-efficacy, stronger 
organization and attention to studies, and class communication than did senior students, 
indicating that freshman may be more excited about learning or have less workload than do 
seniors, which affects their learning effectiveness (2012). Tempelaar, Niculescu, Rientes, 
Gijselaers, and Giesbers (2012) found that goal-setting behaviors have a marginal effect of 
student achievement emotions; however, a student’s effort views substantially impact their 
achievement emotions. 


Cognitive Theories 


Working Memory 

Current frameworks for multimedia learning are largely based on an understanding of human 
working memory, attention, and information processing perspectives. In this section, the 
discussion is limited to learning outcomes rather than perceptual or motivational outcomes. 


Memory Structures and the Cognitive Architecture 


For the purposes of this report, cognitive architecture consists of the sensory memory, working 
memory, and long-term memory (Mayer, 2014a), with the long-term memory being organized 
by structures called schema (Sweller, Ayers, & Kalyuga, 2011). The sensory memory represents 
how information is brought into the cognition space. For the purposes of common distributed 
learning contexts, we move forward with the cognitive theory of multimedia learning model’s 
(Mayer, 2009, 2017) description of the sensory memory, which suggests that the sensory 
memory consists of both the eyes and the ears. There are numerous theories of working 
memory, its structure, and how information moves into the long-term memory. The interaction 
between working memory and attention can be complex (Awh, Vogel, & Oh, 2006). However, it 
is commonly held that the working memory is limited in capacity (Cowan, 2000, 2010; Mayer, 
2014a; Paas & Sweller, 2014). This contrasts with the expansive long-term memory (Paas & 
Sweller, 2014; Sweller et al., 2011), which can maintain copious amounts of information in 
schemas (Sweller et al., 2011). With this basic understanding of the cognitive architecture, 
cognitive theories of learning from both general contexts (cognitive load theory) and multimedia 
learning contexts (the cognitive theory of multimedia learning) become relevant. 


Cognitive load theory (CLT) 


The limitations of human memory when dealing with new information are widely accepted 
(Sweller, Van Merriénboer, & Paas, 1998). Cognitive load theory (CLT) describes how working 
memory can only process a small number of independent elements of information at any one 
time (Baddeley, 1986; Mayer, & Moreno, 1998). Furthermore, CLT stipulates that too many 
elements in working memory impedes cognitive processing (Kalyuga, Chandler, & Sweller, 
1999). Thus, cognitive load theorists suggest that working memory limitations must guide the 
design and presentation of instructional materials (Paas & Sweller, 2014; Sweller 1988, 1989, 
2010; Sweller & Chandler, 1994). To facilitate learning, instructors must provide information ina 
way that allows students to efficiently transition information from the sensory and working 
memory to the long-term memory. Instruction must not overload working memory capacity. 


The implications of the cognitive load theory for instructional design are rather straightforward 
in theory, although potentially more difficult in practice. Instructional designers work to 
moderate the complexity of the learning materials depending upon the prior knowledge, 
experience, and competencies of the learners, and strive to minimize or eliminate sources of 
extraneous cognitive processing to the fullest extent possible (Paas & Sweller, 2014). However, it 
is not always clear what should be considered “extraneous” or ‘““germane” cognitive processing 
for a given task, topic, or learner. In addition, the focus of CLT is on general cognitive processing 
and the theory does not specifically address learning in multimedia or distributed learning 
environments. 


One pitfall of CLT has been in relation to measuring cognitive load. There are numerous 
reported attempts to measure cognitive load through subjective self-report instruments (e.g., 
Anmarkrud, Andresen, & Braten, 2019; Paas, 1992), dual-task methodologies (e.g., Korbach, 
Briinken, & Park, 2018), and physiological measurements (e.g., Dan & Reiner, 2017), however, 
there is little agreement on how to measure it. Specifically, debate continues over whether types 
of cognitive load (i.e., intrinsic and extraneous) can and should be measured or if the 
measurement should only include overall cognitive load. These are important issues because the 
instructional design effects noted in the next section have been posited to occur due to specific 
manipulations of intrinsic and extraneous cognitive load. Accordingly, we describe the effects as 
they are described in the literature, despite the limitations in measurement. For more in-depth 
discussions of the complications encountered when measuring cognitive load, see De Jong, 
(2010), Schroeder and Cenkci (in press), and Zheng (2018). 


Generalizable Design Principles 


Research on Cognitive Load Theory had discovered several instructional design effects. This 
section briefly summarizes each effect derived from CLT. 


The goal-free effect. Providing broader, non-specific instructional goals rather than a discrete 
goal facilitates learning in some situations (Ayres, 1993; Owen & Sweller, 1985; Sweller et al., 
2011). Astudy by Ayers (1993) demonstrated that students in a goal-free group, working under 
the prompt, “find all known angles” outperformed students using the prompt, “find x”. While at 
face value this effect seems to contradict the fundamental tenets of CLT, Sweller et al. argued that 
removing a discrete goal for novice learners can help prevent the use of means-end analysis 
strategies which can increase extraneous cognitive load. 


It appears that this design principle operates under nuanced use cases. For instance, some work 
around the goal-free effect has focused on determining whether the effect is specific to learning 
goals or problem-solving goals. One study found that non-specific goals can facilitate 
instructional efficiency (Kiinsting, Wirth, & Paas, 2011). Further work in the area has shown that 
non-specific problem-solving goals can aid learning and decrease cognitive load, but 
nonspecific learning goals did not significantly influence learning outcomes (Wirth, Kunsting, & 
Leutner, 2009). 


It is important to note that there may be exceptions to this design principle. For example, when 
the goal-free effect has been applied to video game environments, one study found no 
significant differences in learning outcomes (Nebel, Schneider, Schledjewski, & Rey, 2017), 
while another found mixed results (Erhel & Jamet, 2019). Accordingly, instructional designers 
should consider the context of the learning environment when implementing this design 
strategy. 


The worked example effect. Learning can be facilitated by providing worked out examples of 
sample problems for students to study (Booth, McGinn, Young, & Barbieri, 2015; Sweller et al., 
2011). There is an extensive literature base on learning from worked examples. Booth et al.’s 
(2015) review highlights how the effect has been found across a number of domains (primarily 
those involving some sort of mathematics, such as physics or algebra), as well as in the 
laboratory, classroom, and online settings. This is further supported by Renkl’s (2014) review, 
which cites evidence that worked examples are typically efficient and effective for learning. 
Renkl (2014) described how worked examples are typically implemented: first, a theory, 
principle, or instructional strategy is introduced, after which worked examples are provided. 
Once learners have studied these worked examples, they are provided problems to solve 
(Renkl, 2014). 


The worked example effect is thought to occur because when a novice tries to solve an 
unfamiliar problem, they may make guesses about the problem-solving strategy, creating 
sources of extraneous cognitive load (Sweller et al., 2011). Sweller et al. (2011) suggested that 
providing worked examples gives learners the strategies they need to solve the problem, thus 
reducing the extraneous cognitive load imposed by what could otherwise be means-end 
strategy use. 

Booth et al. (2015) discussed four variants of worked examples that have been found to be 
effective, including comparing worked examples, incorrect worked examples, fading worked 
examples, and incorporating self-explanation into worked examples. Meanwhile, Renkl (2014) 
provided 10 instructional design principles for designing worked examples. For brief overviews 
of these strategies, see Booth et al. (2015) and Renkl (2014). 


The split-attention effect. Requiring learners to split their attention between two or more 
sources of information impedes learning, and therefore the salient information should be 
presented in a coherent, integrated format to facilitate deep learning (Sweller et al., 2011). 
According to CLT, the split-attention principle takes place because integrated graphic designs 
lower the extraneous cognitive load imposed by the learning materials (Ayres & Sweller, 2014). 
Specifically, Ayres and Sweller (2014) specified that split designs require learners to integrate 
the two pieces of information in their mind, whereas integrated graphic designs reduce this 


cognitive burden. The inhibiting effects of split-attention have occurred regardless of whether 
attention is split through temporal means, such as when an image is shown and accompanying 
narration is delayed, or spatial means, such as when a diagram and its labels are presented 
separately rather than in an integrated format (Ginns, 2006). The split-attention principle occurs 
when all information is necessary for understanding, when learners have low prior knowledge, 
and when the information being presented is complex (Ayres & Sweller, 2014). 

Concerning temporal split-attention, a meta-analysis of 13 studies has shown that presenting 
relevant words and pictures at the same time can facilitate learning to a large effect (d = .78, 
Ginns, 2006). Similarly, a review of nine studies found a median effect size of d = 1.22 (Mayer & 
Fiorella, 2014). However, the effect may occur to a lesser extent when learners have high-spatial 
ability, when learners have control of the system, and when the information is complex or 
exceeds the working memory capacity (Mayer & Fiorella, 2014). 

Spatial split-attention has received significantly more research than temporal split-attention. Two 
meta-analyses, with the more recent analyzing 58 independent comparisons, found effect sizes 
ranging from moderate (g = .63, Schroeder & Cenkci, 2018) to large (d = .72, Ginns, 2006). A 
review of 22 studies also found a median effect size of d = 1.10 (Mayer & Fiorella, 2014). While a 
few researchers have suggested conditions in which the spatial split-attention effect may not 
occur, Schroeder and Cenkci (2018) concluded that “Across a range of moderator variables, 
integrated designs were in almost all cases found to benefit learning” (p. 698). 


The modality effect. The modality effect suggests that written text that accompanies a graphic 
should be provided through an auditory format rather than written (Mayer & Pilegard, 2014; 
Sweller et al., 2011). Research around the modality effect is extensive and has examined a wide 


variety of potentially moderating variables. Reviews and meta-analyses on the effect have shown 


that this strategy can facilitate learning, with effect sizes including d = .20 (meta-analysis after 
correcting for publication bias, Reinwein, 2012), d = .72 (meta-analysis, Ginns, 2005), and d = .76 
(median effect size, Mayer & Pilegard, 2014). 


The modality effect is believed to work because it reduces the amount of information being 
processed through visual means. As described in the Cognitive Theory of Multimedia Learning 
section, there are thought to be two primary ways of processing multimedia, the visual and 
auditory channels (Mayer, 2009; 2017), a view that is consistent with working memory research 
(Sweller et al., 2011). Researchers have suggested that written text and visual images together 
can overload the working memory capacity, however one can facilitate learning by offloading 
the written information to the auditory channel of working memory, thus reducing the strain on 
the visual channel (Mayer & Pilegard, 2014; Sweller et al., 2011). However, this interpretation has 
been questioned by researchers in the field. Reinwein (2012) argued that the working memory 
model being interpreted through CLT and the Cognitive Theory of Multimedia Learning cannot 
be used to explain the effect. A full explanation of this argument is outside of the scope of this 
paper. For more information see Reinwein (2012). 


There are conditions in which the modality effect may not occur or may even be reversed. For 
instance, there is an on-going debate as to whether the pacing of learning materials can 
influence the effect, with some research showing that system-paced materials have the largest 
benefits and learner-paced materials showing an inverse effect (Ginns, 2005; Mayer & Pilegard, 
2014). Meanwhile, Sweller et al. (2011) argued that the modality effect was robust across pacing 
conditions. Reinwein (2012) also found that the type of visualization can influence the effect, with 
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dynamic visualizations benefiting from the effect more than static. Other moderating variables 
include the complexity of the material, the learners’ prior knowledge, the length of the material, 
and the learners’ familiarity with the terms used in the instructional materials (Mayer & Pilegard, 
2014). 


The redundancy effect. The redundancy effect suggests that two sources of information that 
convey the same material, when presented simultaneously, can interfere with learning 
compared to if only a single source is used (Kalyuga & Sweller, 2014; Mayer & Fiorella, 2014; 
Sweller et al., 2011). For example, if a process diagram explains all of the processes within the 
diagram itself, but is accompanied by a redundant paragraph that adds no new information, it 
may not benefit learning because the diagram or the text could be understood by itself. 
Similarly, there are other situations in which redundancy can occur, such as when extraneous 
information is presented, but is unnecessary (Kalyuga & Sweller, 2014). This situation is also 
known as the coherence effect (Kalyuga & Sweller, 2014; Mayer & Fiorella, 2014). 

The redundancy effect is thought to occur because it reduces extraneous cognitive load (Kalyuga 
& Sweller, 2014; Sweller et al., 2011). Research has shown that non-redundant presentations can 
benefit learning, with a review finding a median effect size of d = .86 (Mayer & Fiorella, 2014). It 
is also noteworthy that there are situations where the redundancy effect does not occur or may 
reverse. Mayer and Fiorella (2014) outlined these situations when a reverse effect may be seen, 
such as when only a few words of on-screen text are used, when learners have high prior 
knowledge, and when there are no graphics used in the presentation. 


The expertise reversal effect. The expertise reversal effect suggests that in some situations, 
learners with more expertise may not benefit as much, or at all, from instructional design 
strategies that benefit more inexperienced or less knowledgeable learners (Sweller et al., 2011). 
For example, Blayney, Kalyuga, and Sweller (2015) described how strong instructional supports 
are generally beneficial for low-experience learners, while the same supports may interfere with 
more knowledgeable learners’ ability to learn the information. Recently, Chen, Kalyuga, and 
Sweller (2017) argued that this effect was a subset of the element interactivity effect, as 
described later in this report. 


The guidance fading effect. Briefly, the guidance fading effect suggests that, as learners gain 
experience and knowledge to solve problems, the guidance and scaffolding should be faded in 
order to promote more problem-solving rather than guidance-following (Chen, Retnowati, & 
Kalyuga, 2019; Sweller et al., 2011). This effect is related to the redundancy effect and the 
expertise reversal effect. Essentially, when moving from worked examples to student-driven 
problem-solving tasks, the steps in the worked examples that are provided should be gradually 
removed, so the student solves more of the problem on their own over time (Chen et al., 2019). 
This process aims to remove extraneous cognitive load that may be due to redundancy as the 
learner gains proficiency at solving problems (Chen et al., 2019). 


The imagination effect. Sweller et al. (2011) found the imagination effect suggests that, for 
learners with more experience in a domain, imagining how to solve the problem can aid 
learning compared to studying another worked example, which may be redundant. Presumably, 
allocating the working memory resources towards imagining how to solve the problem prevents 


working memory resources from being directed towards studying a redundant worked 


example. However, as noted by Sweller et al. (2011), there are only certain conditions in which 
the effect may occur (Sweller et al., 2011). 


The self-explanation effect. Sweller et al. (2011) described the self-explanation effect as 
related to the imagination effect in that, through self-explanation, learners are asked to think 
about how a problem might be solved and why, which requires the learner to think about the 
relations between different pieces of information. As with the imagination effect, this effect is 
more effective for learners with experience in the domain rather than novice learners (Sweller et 
al., 2011). 


The element interactivity effect. Simply stated, the element interactivity effect suggests that 
the instructional design effects outlined here may not occur if the learning materials are not 
complex (Chen et al., 2017; Sweller et al., 2011). In short, Sweller et al. (2011) suggested that 
when learning materials are not complex, the instructional design strategies used may not matter 
as much because it may not overload the learners’ working memory capacity. 


The transient information effect. The transient information effect occurs when information is 
presented to a learner in such a way that it is gone before the learner can process it all (Leahy & 
Sweller, 2016; Sweller et al., 2011). Sweller et al. (2011) provided an example of a lecture - an 
instructor may verbally provide a lot of complex and interacting information that must be 
learned, but unless that information is written down or presented elsewhere, the learner may not 
process all of it in time before the teacher moves to the next topic, which could impede learning. 
Sweller et al. (2011) suggested that the transient information effect may explain the reverse 
modality effect findings (discussed in the modality effect section). 


The collective working memory effect. Sweller et al. (2011) summarized the collective 


working memory effect as learners can learn more through collaborating with others than 


working alone, but the effect typically only occurs when the information is complex. When the 
information is not complex, individual learning seems to produce better results. This is a 
relatively new area of inquiry and there is limited research in the area compared to some other 
cognitive load theory effects. (Sweller et al., 2011) 


Blended Learning and CLT 


Transitioning a face-to-face course to a blended format may become increasingly common as 
blended learning becomes more prevalent. Designers and instructors will need to ensure that 
learners maximize their prior knowledge and motivation as more online instruction is utilized 
(Impelluso, 2009). Impelluso (2009) redesigned a computing course for non-majors using a 
blended learning format to maximize the germane cognitive load and reduce the extraneous 
cognitive load. To achieve this transition, vertical and horizontal scaffolding was implemented 
using a blended format that was half online and half face-to-face. Impelluso (2009) reported that 
using the blended format enhanced and streamlined the delivery of course materials, improved 
instructor evaluations, provided a cost savings to the institution, improved learning outcomes, 
and improved enrollment rates for the class (Impelluso, 2009). 


Mattis (2015) found that a flipped classroom format afforded a decrease in mental effort and an 
increase in accuracy in the student cohort as problem complexity increased. The difference was 
significant between the two delivery modes in problems of moderate complexity and the 
modality effect was absent as accuracy improved to some degree at all levels of problem 


complexity (Mattis, 2015). Similarly, Akkaraju (2016) found that threshold concepts in physiology 
learning could be improved as demonstrated by higher retention and pass rates. Akkaraju 
stated that intrinsic cognitive load was addressed in the flipped classroom by utilizing retrieval 
practice and pre-training. Extraneous cognitive load was ameliorated by using intentional 
content and germane cognitive load was increased by using increasing problem-solving time 
during face-to-face meetings. 


Students’ and tutors’ reactions to a flipped classroom environment can be mixed (Goedhart, 
Blignaut-van Westrhenen, Moser, & Zweekhorst, 2019). Goedhart et al. (2019) found that while 
both groups had positive responses to the overall blended experience, praising the 
personalized pre-class learning and peer-learning classroom activities, some students did not 
agree that the flipped format contributed to positive learning outcomes. Students in the flipped 
classroom were shown to come to classroom meetings more prepared than for traditional 
classroom lectures and the availability of pre-class videos and reading materials gave students 
self-regulation opportunities, which could improve the quality of study (Goedhart et al., 2019). 
Goedhart et al. (2019) posited that providing pre-class content can be beneficial in lowering the 
students’ cognitive load because students can exert learner control over pacing and frequency. 
Mooring, Mitchell, and Burrows (2016) demonstrated that there was a statistically significant 
increase in students’ intellectual accessibility and emotional satisfaction in the blended 
classroom compared to traditional lecture courses as assessed via the Attitude toward the 
Subject of Chemistry Inventory Version 2. Mooring et al. (2016) suggested that their positive 
findings are due to the increased access to class material and the subsequent better use of 
classroom meetings for group work and discussion. This in-class change of format is suggestive 
of a decrease in cognitive load and a consequent increase in the emotional satisfaction and 
intellectual accessibility of the learners. Further, their findings encourage the use of blended 
learning in large enrollment, challenging courses (Mooring et al., 2016). 


Cognitive theory of multimedia learning (CTML) 


Mayer (2009, 2017) has defined a theory for multimedia learning that builds on core cognitive 
mechanisms applied specifically to multimedia materials. Mayer (2009, 2017) defines 
multimedia learning as learning via both verbal and pictorial representations (e.g., text, labels, 
diagrams, animations, and videos). Like CLT, CTML assumes that the working memory has a 
limited capacity, but the framework also includes two other assumptions: 1) that learners have 
two processors for handling new information depending on their modality (an auditory 
processor and a visual processor), and 2) that learners must be cognitively engaged to produce 
new knowledge structures based on these inputs (Mayer 2014). This cognitive engagement is 
viewed as processes of selecting relevant information, organizing it, and integrating it with their 
prior knowledge structures. 


Regarding the first assumption, Mayer (2009) articulated how information is brought into 
working memory through either the eyes or the ears. Importantly, these separate processors can 
work together to create more concrete mental models. For example, in a multimedia lesson 
about bird flight, a learner might be exposed to the video of a bird in motion (visual) with 
narrated explanations of the process (auditory). A mental model of the process can be formed 
based on either the visual or auditory information. Yet, together, the visual and auditory mental 


models can result in a more elaborate, and potentially more accurate, schema than information 
presented to either single processor alone. 


Also, important, however, is that each processor is limited in capacity (Mayer, 2014), which 
echoes the limitations of working memory capacity as denoted in CLT. Thus, too much 
information presented to any one modality can overwhelm that processor. However, 
instructional designers, being aware of this fact, can strategically present information in both 
modalities to prevent such overload. 


Regarding the second assumption, developing a coherent schema using both modalities takes 
work. Learners must actively put forth effort in their cognitive processes to learn. This active 
processing is described as consisting of selecting relevant words and pictures, organizing 
relevant words and pictures, and then integrating them into coherent mental models (Mayer, 
2014a). 


To summarize, CTML has similar implications as CLT but provides additional guidance for 
learning in multimedia learning environments. Since CTML describes two processors when 
dealing with multimedia learning materials, one can infer that designers need to moderate the 
amount of essential processing required by the learners depending upon their prior knowledge, 
experience, and competencies by moderating the amount of information sent through each 
processor. Furthermore, we can simultaneously present complementary information through 
both channels to create more elaborate mental models. 


Generalizable Design Principles 


Based on research around the CTML, researchers have investigated a variety of design 
principles derived from this work, including (Mayer, 2014b, 2017): 


Coherence effect. See ‘redundancy effect’ in the cognitive load theory section. 


Signaling effect. The signaling effect suggests that adding cues to direct a learner’s attention can 
facilitate learning (Mayer & Fiorella, 2014). A review of the signaling principle found a median 
effect size of d =. 41 (Mayer & Fiorella, 2014), while a larger meta-analysis found an overall effect 
size of r= .17 (Richter, Scheiter, & Eitel, 2016). Meanwhile, Schneider, Beege, Nebel, and Rey’s 
(2018) meta-analysis of the signaling effect found effect sizes of g+ = .53 (retention) and g+ = .33 
(transfer). Overall, signaling can benefit learning in some situations. 


Signaling is believed to be an effective instructional design strategy because it reduces 
extraneous processing (Mayer & Fiorella, 2014). Specifically, when a complex graphic is placed 
on the screen, and a learner is asked to find a specific component in the graphic in order to 
understand the system, they may need to engage in a visual search task to find the relevant 
component, and this visual search task is extraneous cognitive processing. This process can be 
simplified by providing a visual cue that highlights the relevant component. 


Redundancy effect. See ‘redundancy effect’ in the cognitive load theory section. 
Spatial contiguity effect. See ‘split-attention effect’ in the cognitive load theory section. 
Temporal contiguity effect. See ‘split-attention effect’ in the cognitive load theory section. 


Segmenting effect. Mayer and Pilegard (2014) suggested that verbal information, such as 
narration during an instructional video, should be segmented into 8-10-second-long segments to 
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facilitate learning. This is known as the segmenting effect. Presumably this is effective as it helps 
manage the amount of essential processing the learner is engaging in at any given time, and 
allows them the ability to fully process the information before deciding to move to the next 
segment (Mayer & Pilegard, 2014). In their review, Mayer and Pilegard found a median effect 
size of d= .79. 


It is noteworthy that in practice, rather than in a research lab, breaking instruction into 8-10 
second segments may not be feasible. Exploring ways around this constraint, Schroeder, Chin, 
and Craig (in press) tested the effects of a segmented video, a non-segmented video with no 
learner control, and a non-segmented video with learner control. Their results indicated that 
giving the learner some form of control over the instructional video (such as pause, rewind, and 
fast forward) may be a feasible way of providing similar benefits as physically segmenting the 
video. 


Pre-training effect. The pre-training effect states that “people learn more deeply from a 
multimedia message when they know the names and characteristics of the main concepts” 
(Mayer & Pilegard, 2014). Specifically, Mayer and Pilegard (2014) suggested that presenting the 
main ideas before describing how they are related to one another can benefit learning. For 
example, if a student is learning about how a bird flies, it may be beneficial to first learn about 
the shape of a bird’s wing and how air moves around shapes before discussing how air moves 
around the shape of the bird’s wing. Mayer and Pilegard’s (2014) review of studies around this 
principle found a median effect size of d = .75. Presumably, pre-training works because it 
reduces the amount of essential processing that the learner must engage in at any given time, 
thus helping to prevent overloading the working memory capacity (Mayer & Pilegard, 2014). 


Modality. See ‘modality effect’ in the cognitive load theory section. 


Personalization effect. The personalization effect suggests that using a conversational style for 
text or narration rather than a formal style can benefit learning (Mayer, 2014c). For example, 
instead of saying, “The heart pumps the blood through the human body,” one could instead say, 
“Your heart pumps the blood through your body.” Presumably, this simple change in wording 
helps establish the social processes in the learner (see social agency theory elsewhere in this 
report), which can lead to deeper learning (Mayer, 2014c). In his review, Mayer (2014c) found a 
median effect size of d = .79. Meanwhile, Ginns, Martin, and Marsh’s (2013) meta-analysis of the 
effect found that conversational style text aided both retention (d = .30) and transfer (d = .54). 


Voice effect. The voice effect suggests that ‘people learn more deeply when narration in a 
multimedia lesson is spoken in a standard-accented human voice rather than in a machine voice” 
(Mayer, 2014b), and Mayer (2014b) found a strong median effect size supporting his review (d = 
.14). However, Mayer’s review is based on only six comparisons, with four of the six studies 
being conducted more than a decade ago. 


More recently, the question of voice type in narration has been re-examined by researchers, 
often in the context of virtual humans. Craig and Schroeder (2017; 2019) argued that the voice 
effect could be an artifact of the technologies available when the original research was 
conducted. Craig and Schroeder (2017) found that a modern text-to-speech generated voice 
paired with a virtual human was more effective for learning on transfer outcomes and 
instructional efficiency than an old text-to-speech engine and a recorded human voice. 


Meanwhile, Craig and Schroeder (2019) examined the same three voice conditions without an 
agent and found largely no significant differences among groups. 


Taking a different approach, Davis, Vincent, and Park (2019) argued that the discussion about 
what voice to use should move beyond simple human or computer discussions and suggested 
that voice prosody could be an important factor to consider. In their study with non-native 
language speakers, they found no significant differences in learning outcomes between those 
who learned with high or low prosodic voices or a computerized voice (Davis, Vincent, & Park, 
2019). 


In summary, research around the voice effect has shown mixed results. At the present time, it is 
not abundantly clear whether recorded human voices or modern text-to-speech engines are 
more effective for facilitating learning. Moreover, research is needed to understand what voice 
features influence learning, as well as learners’ perceptions of the voice. 


Image effect. The image effect suggests that adding the visual image of the speaker on the 
screen may not greatly benefit learning (Mayer, 2014b), however this is a point of debate among 
scholars. In his review, Mayer notes that considered studies generally used speaking characters 
that were of low embodiment, meaning that they were not very humanlike, were stationary, or 
may not have used effective gestures. Overall, Mayer found a median effect size of d = .20,a 
small effect. This is consistent with the findings of Schroeder, Adesope, and Gilbert’s (2013) 
meta-analysis, which found an overall impact of the use of virtual characters to be g = .19. 
Similarly, Heidig and Clarebout’s (2011) systematic review found that there were largely no 
significant differences between groups that learned with or without a virtual character on the 
screen. However, all three sets of researchers noted that questions of the mere inclusion of a 
virtual teacher is too broad, and researchers should focus on the design of the virtual teacher 
rather than its inclusion. Heidig and Clarebout specifically highlighted potential design 
considerations, while Schroeder et al. revealed in which contexts the virtual on-screen 
characters show effectiveness. The most recent work in this area was Craig and Schroeder’s 
(2018) review, which documents specific design considerations. 


To summarize, research has shown that including an image of the teacher as a virtual character 
can be effective in some use cases. However, more research is needed to understand the 
implementations and populations for which they are most effective. 


Embodiment effect. The embodiment effect suggests that on-screen characters that are 
embodied through gestures, such as pointing to relevant content, or have facial expressions, can 
increase learning compared to on-screen characters with less embodiment (Mayer, 2014b). 
Through his review, Mayer (2014b) found a median effect size of d = .36 across 11 comparisons. 


In a more focused review and meta-analysis, Davis (2018) examined the influence of on-screen 


characters’ gestures on learning outcomes. He found that gesturing on-screen characters 
facilitated both retention (g = .28) and near transfer (g = .39). However, Davis also highlighted a 
limitation in the sample he analyzed, which was that most studies only included signaling 
gestures (“deictic gestures to direct spatial awareness” (p. 204)). Overall, the research around 
the embodiment effect is somewhat limited in scope. While there has been a fair number of 


studies investigating how on-screen characters’ gestures influence learning, many of these 
studies have been focused on gestures used by virtual characters rather than on-screen humans 
(e.g., Davis, 2018), and many studies have used signaling gestures rather than other types of 


gestures. Clearly, more research is needed to more clearly understand when and what types of 
gestures should be used, and if the gestures used by actual human actors should be different 
than those used by virtual characters. 


Guided discovery effect. It is well known that discovery learning, presented without any sort of 
guidance, generally is not effective when compared to other instructional strategies (De Jong & 
Lazonder, 2014). However, de Jong and Lazonder (2014) argued that guided discovery learning 
can be effective if the guidance is suitable for the learners. They highlighted six different types 
of guidance that can be incorporated into discovery learning environments: process constraints, 
performance dashboard, prompts, heuristics, scaffolds, and direct instruction. Importantly, de 
Jong and Lazonder noted that “As research on the orchestration of different types of guidance is 
still in its infancy, there are no concrete recommendations on this issue yet, meaning that 
teachers and instructional designers must rely on their professional insights” (p. 384). 


Worked examples. See ‘worked example effect’ in the cognitive load theory section. 
Self-explanation. See ‘self-explanation effect’ in the cognitive load theory section. 


Generative drawing effect. The generative drawing effect suggests that having learners draw 
pictures during the learning process can aid learning because it facilitates generative 
processing (Leutner & Schmeck, 2014). However, Leutner and Schmeck (2014) noted that it is 
important that the drawing task itself is not designed in such a way that it induces extraneous 
processing. For example, they suggested that extraneous processing can be minimized by 
providing sample images of the primary items to be drawn, that way the learner does not have to 
create them from scratch and can simply replicate the images into whatever format they need to 
convey. In their review, Leutner and Schmeck (2014) found effect sizes ranging from d = -.16 to d 
= .87 for comprehension items and d = -.05 to d = .90 for transfer tasks. 


Feedback effect. The feedback effect suggests that providing explanatory feedback can benefit 
novice learners more than simple corrective feedback (Johnson & Priest, 2014). Johnson and 
Priest (2014) specified that explanatory feedback refers to feedback which explains why an 
answer was right or wrong based on an underlying theory, mechanism, or principle. They 
suggested that corrective feedback creates extraneous processing because the learner must try 
and figure out why their answer was or was not correct, and this process may not benefit 
learning. In Johnson and Priest’s (2014) review of eight comparisons, they found a median effect 
size of d = .72. Finally, in order for explanatory feedback to be most effective, Johnson and Priest 
(2014) suggested that the feedback should facilitate the learner’s active processing, that the 
feedback should be provided in such a way that it is guided by other instructional design 
principles and effects, and that individual learner differences are considered (johnson & Priest, 
2014). 


Multiple representations effect. When providing instruction is it common to represent 
phenomena in more than one way. For example, if learning about how a bird flies, one may 
present an image of a bird’s wing shape, an image showing how air moves around that shape, 
and the mathematical equations that dictate the lift achieved. Ainsworth (2014) suggested that 
multiple representations can be used in three ways: complementing one another, constraining 
something that is inherently complex to make it easier to understand, or for helping a learner 
construct a deeper understanding of the topic at hand. While Ainsworth (2014) provided a 


detailed analysis of each of these use cases, there are also implications for instructional design 
provided. Namely, Ainswoth (2014) described how designers should try to minimize the number 
of representations used in order to facilitate reaching the learning goal, and when multiple 
representations are used, the learner should be encouraged to examine the representations in 


relation to one another. 


Learner control effect. It is a common practice to provide learners with control over their 
computer-based learning environment. However, the term learner control can mean different 
things to different people. For the purposes of describing this effect, Scheiter (2014) described 
learner control as being related to the pacing and sequencing of the learning materials, as well 
as the selection of those materials and the way they are displayed. When examined in this light, 
it is apparent that decisions on how much control over their computer-based learning 
environment to provide learners can be complex. The learner control effect suggests that 
providing learners control over their learning environment can be effective, but only when 
supplemental scaffolding is provided and learners have high prior knowledge (Scheiter, 2014). 
As Scheiter (2014) summarized, there are limited situations in which research has shown learner 
control, as defined here, to be beneficial for learning. 


It is important to note that learner control can be viewed as a continuum from no learner control 
through full learner control, as described in the preceding paragraph (Scheiter, 2014; Schroeder 
et al., in press). As such, it is important to realize that providing some aspects of learner control 
may be helpful in some situations. For example, Schroeder et al. (in press) found that having 
control over the pace of the instructional video aided learning, and some of the research around 
the segmenting principle is based on learner-controlled segments. 


Collaboration effect. Collaboration is a common aspect of many active learning courses. 
However, there are certain situations in which collaboration is most beneficial. Kirschner, 
Kirschner, and Janssen (2014) suggested that in multimedia learning, collaboration is effective 
when the environment provides the proper scaffolding and tools to facilitate the interactions, 
when the information and cognitive tasks are distributed among group members, and when the 
task is sufficiently complex to require collaboration (see the collective working memory effect). 
To summarize, collaborative tasks should be intentionally designed to facilitate learning. Simple 
collaboration for the sake of collaboration will not always facilitate learning. Table 1 delineates 
their recommendations. However, instructional designers seeking to create collaborative 
learning environments are encouraged to see Kirchner et al.’s (2014) extensive review. 


Table 1. Asummary of Kirchner et al.’s (2014) recommendations for collaborative multimedia learning environments. 


Collaborative tasks need to be cognitively 

demanding 1. Collaborative tasks should 
require more information processing 
than one individual could handle 
efficiently on their own. 
2. Tasks must be adjusted as 
learners gain expertise. 


Scaffolding should be provided by the 

environment 1. Provide the learners with a 
collaborative workspace. 
2. Ensure learners are familiar 
with other team members areas of 
expertise. 
3. Learners should be required to 
evaluate their contribution to the team 
regarding their efforts invested. 
4. The environment should allow 
for varied inputs and allow for the 
relations between these inputs to be 
shown. 
5. Ensure all group members 
have access to all the materials. 
6. Ensure the entire team 
understands the end goal of the task. 


Distribution of information and tasks among 


the team 1. Provide channels for 
communication. 
2. Ensure team members know 
what the other team members are 
working on.3. 
3. Promote group awareness with 
respect to how the team is functioning 
and who is working on what task. 


Expertise reversal. See ‘expertise reversal effect’ in the cognitive load theory section. 


Animations. Research in the multimedia learning space has also provided specific guidance for 
the design of animations. Animations differ from videos because videos capture everything 
within the scene, regardless of whether it is relevant to the learning situation, whereas 
animations are intentionally designed from the ground up and therefore can be made 100% 
relevant to the learning situation (Lowe & Schnotz, 2014). Arecent meta-analysis showed that 
animations were effective compared to static graphics (g = .23, p < .001, Berney & Bétrancourt, 
2016). Yet, the question of animation design is more complex than simply “creating an 
animation.” Lowe and Schnotz (2014) recommended the following for designing effective 
multimedia animations: a) define the animation’s purpose before design so that it can be 
purposefully designed, b) carefully examine the needs of the learning situation and the 
affordances of the medium chosen. In short, animation is not always the best medium, c) 
consider how the animation will be perceived by the learner, and ensure that the perceptual 
attributes of the animation facilitate the desired goals, d) provide cueing when necessary within 
the animation, but keep in mind that traditional cues may not be as effective in animations due to 
the medium, and alternatives may need to be considered, and e) interactive animations can be 
effective, however the designer must consider the design of interactivity with the same level of 
attention as the animation itself. 


Blended Learning and CTML 


Because blended learning courses generally offer multimedia components for students to view 
online, the CTML must be in the forefront of media development (Amaka & Goeman, 2017). A 
review of the literature by Amaka and Goeman (2017) revealed that several design principles 
should govern multimedia use in blended courses, namely, navigability, flexibility, interactivity, 
(a)synchronicity, ease of use, media richness, individualization, mobility, proximity, and 
responsiveness. They found that the studies that were reviewed supported the CTML and the 
Media Richness Theory (MRT), and that the multimedia principle was upheld as most studies 
demonstrated that media fostered deeper learning. Many of the studies were U.S. higher 
education based and the authors caution that the trends may not be easily transferred to other 
locations and populations. 


The Role of Affect in Learning 


In addition to beliefs about one’s capabilities, students also experience meaningful emotions 
related to the learning experience or environment. In contrast to classic emotions described by 
Ekman and colleagues (e.g., anger, disgust, happiness, etc.), academic emotions can be 
described as cognitive-affective states that represent a cognitive experience (e.g., failing to 
understand a topic) and an emotional response to that experience (e.g., confusion or frustration; 
Ekman, Friesen, & Ellsworth, 1972,). These emotions have been shown to impact self-regulation 
(Pekrun, Goetz, Titz, & Perry, 2002), engagement (Pekrun & Linnenbrink-Garcia, 2012), and 
academic achievement (Pekrun et al., 2002; D’Mello & Graeser, 2012). 


However, academic emotions can have complicated relationships with learning. The learner’s 
prior knowledge has been shown to predict enjoyment of future learning (Goetz, Frenzel, Hall, & 
Pekrun, 2008). Further, the lack of enjoyment of the learning experience (e.g., boredom) has 
shown negative relationships to learning (Baker, D’Mello, & Rodrigo, 2010; Craig, et al., 2004). 
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Emotions are prevalent when working with technology (D’Mello, 2013) and have been shown to 
have a significant impact on learning (D’Mello, 2013; D’Mello & Graesser, 2012). Research on 
affect and technology has mainly focused on detection and creating systems that can 
dynamically respond. For example, an effective, affect-sensitive version of the tutoring system 
AutoTutor included a pedagogical agent that was responsive to the learner’s affective state and 
displayed affective expression and feedback (D’Mello, Craig, Fike, & Graesser, 2009; D’Mello & 
Graeser, 2008). However, it has also been shown that agent displaying emotions such as 
uncertainty or skepticism within non-interactive environments can have a small positive impact 
on learning (Sullins, Craig, & Graesser, 2009). 


In the multimedia learning literature, affective or emotional response is a relatively new line of 
inquiry. However, multiple potential models have been proposed by researchers, such as the 
Cognitive-Affective Theory of Learning with Media (CATLM, Moreno, 2006; Moreno & Mayer, 
2007) and the Integrated Cognitive Affective Model of Learning with Multimedia (ICALM, Plass & 
Kaplan, 2016), which have been influential in research around emotional multimedia design. We 
review each of these areas of research below, in addition to the Control Value Theory of 
Achievement Emotions (CVTAE, Pekrun, 2006). 


Cognitive-Affective Theory of Learning with Media 

The CATLM is an extension of the CTML (Mayer, 2014; as discussed elsewhere in this report). In 
short, the CATLM extends the CTML by modeling how all five senses of humans are part of the 
sensory memory rather than only sight and hearing (as is the case in CTML), and noting that 
metacognitive processes, motivation, and affect can influence attention and perception, as well 
as the selection, organization, and integration of learning material in the working and long-term 
memories (Moreno, 2006; Moreno & Mayer, 2007). 


Integrated Cognitive Affective Model of Learning with Multimedia 


The ICALM can be viewed as a modification of the CATLM or an extension of the CTML. 
Whereas CATLM notes meta-cognitive processes, motivation, and affect as influencing the 
learning process, ICALM suggests that affect is sensed from the learning materials, and that 
emotional self-regulation, in the form of attributed affect or mood, and then interest, motivation, 
or mood influence the organization and integration of information in the working memory (Plass 
& Kaplan, 2016). 


Emotional Design 


Relatedly, researchers in the multimedia learning literature have explored the notion of 
“emotional design’, which seeks to encourage positive emotions from multimedia learning 
environments (Um, Plass, Hayward, & Homer, 2012). These types of studies have examined how 
features such as shapes (Plass, Heidig, Hayward, Homer, & Um, 2014; Um et al., 2012), colors 
(Plass et al., 2014; Um et al., 2012), aesthetics (Heidig, Miller, & Reichelt, 2015), usability, 
emotionally-charged and context-specific pictures (Schneider, Nebel, & Rey, 2016), emotionally- 
charged text (Stark, Briinken, & Park, 2018) anthropomorphisms (Park, Knoérzer, Plass, & Briiken, 
2015), and combinations of some of these factors (Mayer & Estrella, 2014) can influence learners’ 
emotions and/or learning outcomes. Leutner (2014) noted that while researchers have sought to 
find if affect mediates learning, in line with CATLM (Moreno, 2006), research in the area is in its 
infancy. However, affective and cognitive trait constructs may influence learning (Leutner, 2014). 
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Control Value Theory of Achievement Emotions 


Researchers have long sought to understand the role of emotions in relation to achievement. 
Pekrun (2006) defines achievement emotions as those “tied directly to achievement activities or 
achievement outcomes” (p. 317), or more specifically, activity emotions and outcome emotions, 
respectively. Pekrun suggested that emotions can be about a specific point in time, known as 
state achievement emotions, or emotions that are experienced repeatedly in reference to a 
specific task or outcome, known as trait achievement emotions. 


At the high level, Pekrun (2006) outlined how value (positive, negative, or mix) and control (from 
low to high, or internal, external, or irrelevant) appraisals can focus on activities, or prospective 
or retrospective outcomes (p. 320). Moreover, Pekrun highlighted specific emotions tied to these 
appraisals and focuses, including hope, joy, shame, frustration, and boredom, among others. 
Finally, Pekrun noted that these emotions are likely domain specific, like how self-concepts are 
domain specific. 


There are several determinants of achievement emotions, including, but not limited to factors 
such as achievement goals, social and cultural factors, cognitive resources, and self-regulation 
(Pekrun, 2006). Perhaps most importantly, Pekrun (2006) noted how the design of the learning 
environment and the social environment can influence appraisals, emotions, and subsequently, 
learning outcomes (p. 328). 


While the influence of emotions on learning processes is of interest to many educators, there are 
challenges associated with measurement. As Pekrun (2006) noted, “the close conceptual and 
empirical links between the three categories represent a challenge for empirical research that 
cannot be sufficiently met to date” (p. 330). However, Pekrun (2006) highlighted the following 
seven considerations for instructional design: 1) the demands of the task, 2) the development of 
values, 3) promoting self-regulation and cooperation, 4) the structure of the goals, 5) the type of 
feedback provided, 6) addressing negative appraisals or emotions that students may have, and 
1) promoting emotional self-regulation (p. 334-336) (Pekrun, 2006). 


Blended Learning and Affect 


Many studies attempt to qualify online course success by analyzing LMS data to examine time 
spent on material and numbers of assignments submitted as predictors of student success; 
however, numbers alone do not tell the whole story of online learning environments (Ramirez- 
Arellano, Bory-Reyes, & Hernandez-Simon, 2019). Ina study of a blended learning environment, 
student performance, as determined by overall grade, was negatively affected by negative 
emotions and test anxiety; however, positive emotions were not linked to improved overall 
grades, suggesting that negative emotions play a stronger role in student performance than do 
positive emotions (Ramirez-Arellano et al., 2019). Jeong, Gonzalez-Gomez, and Canada-Canada 
(2016) found that flipped classrooms elicited more positive student emotions than negative ones 
and demonstrates that blended learning holds promise for use as a learning modality regarding 
student affect. 


Affect in Learning with M00Cs 


Affect in learning plays a significant role in the use of MOOCs; from subtle differences MOOCs 
present in addressing learners in an online environment (Riehemann & Jucks, 2018), to the 
broader studies regarding their overall experiences in the MOOCs (Chang, Yu, & Chun, 2015). 
Drop-out ratio continues to be an issue for MOOCs and affect in learning has been examined in 
attempts to reduce this problem (Wang, He, Guo, & Wu, 2019). The current belief being that the 
drop-out rate could be curbed if MOOCs provided a more positive experience to the user. The 
wide variety of learners coming to the MOOCs systems has also spurred research into 
differences in: attentional levels and cognitive styles (Chang, Lin, & Chen, 2019), goal setting 
(Henderikx, Kreijns, Castafio Mufioz, & Kalz, 2019), and what causes immediate positive or 
negative feelings towards the MOOCs (Xing, Tang, & Pei, 2019). 


Cross-cutting theories /ideas 


Self-regulation, metacognitive monitoring, and regulation 


Detinition and History 

Panadero (2017) analyzed and compared six models of Self-Regulated Learning (SRL). The first 
three models of SRL were developed by Zimmerman from a social-cognitive perspective. The 
first model, the Triadic Analysis of SRL, focuses on the interactions between the environment, the 
person, and behavior and is coordinated with Bandura’s model of social cognition. The second 
model shows the three cyclical phases of SRL including forethought, performance, and self- 
reflection. This model is the initial appearance of the interrelationship of metacognitive and 
motivational processes in SRL. The third model is a modification of the second model, in which 
several subprocesses were added into each cyclical phase of SRL. This last model became a 
representation of Zimmerman’s SRL model. This third model incorporates forethought, 
performance, and self-reflection (Panadero, 2017). The forethought phase represents students’ 
analysis and plans of tasks and goals and the performance phase is the execution of the task and 
plan. Students need to monitor their processes and motivate themselves throughout the phases 
and the self-reflection phase is when students assess their own performance of the task and 
make meaningful reasoning for their success or failure. 


The empirical studies in terms of the effect of this model showed that higher achievers showed 
more use of subprocesses from the cyclical model (DiBenedetto & Zimmerman, 2010); young 
male experts performed more SRL actions compared to young male novices in basketball 
(Cleary & Zimmerman, 2001). 


The next two models of SRL were developed by Boekaerts (1991) from the lens of goal setting in 
relation to emotion. One is named the six-component model of SRL, and another is the dual 
processing self-regulation model. Specifically, the six-component model of SRL is used mainly to 
train teachers and construct new instruments for research due to its well-defined structures 
(Panadero, 2017). The dual processing self-regulation model, as the representation of Boekaerts’ 
work, underscored the role of goal paths in students’ behavior change. In particular, the 
appraisals are crucial to determine students’ goal paths. For example, students were most likely 
to go for the well-being pathway to protect their beliefs (e.g., self-concept of ability) if the task 
triggered their negative emotions. However, if the task is coherent with their needs and goals, 
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their positive emotions were triggered and this would lead to a growth pathway in which 
students would amplify their competence (Boekaerts & Cascallar, 2006; Boekaerts, 2011). 


The empirical support of the dual processing model indicates that competence and value 
appraisals positively influenced students’ outcome assessment and reported effort. For example, 
students who had reported that they were capable of doing their homework turn out to invest 
more effort in doing math homework because they produce more positive emotions during the 
task (Boekaerts, Otten, & Voeten, 2003; Boekaerts & Rozendaal, 2007). Another major 
contribution that Boekaerts (1999) made is the development of the Online Motivation 
Questionnaire (OMQ), which measures the “sensitivity to learn in concrete situations.” For 
instance, on this questionnaire, students self-report their perceptions regarding a certain task 
(e.g., feeling about this task and the effort they will spend on this task). Then they report how 
they evaluate their feelings and attributions after the task (Boekaerts, 2002). 


A widely used model in the computer-supported learning settings (Panadero, Klug, & Jarvela, 
2016), was developed by Winne and Hadwin (1998) by integrating the metacognitive 
perspective. Notably, instead of emphasizing the role of emotion, such as exists in the Dual 
Processing Model, this model only mentions motivation. This model describes a feedback loop 
in which students cognitively monitor their activities and change strategies (Winne & Hadwin, 
1998). There are four phases within this model: a) task definitions (e.g., the understanding of a 
concrete task), b) goal setting and planning (e.g., goals and a plan to achieve the task), c) 
enacting study tactics (e.g., the use of strategies to attain the goals), and d) metacognitively 
adapting studying (e.g., a volitionally long-term and deep changes in motivations, beliefs and 
strategies for the future after the completion of the previous phases) 


Winne and colleagues have more recently explored the effect of this model in computer- 
supported learning environments. They found that tracing students' online learning data brought 
promising insights to the SRL field and provided a new approach to measure SRL (Winne, 
Hadwin, & Gress, 2010; Winne & Hadwin, 2013; Winne & Baker, 2013). For example, Azevedo 
and Hadwin (2005) found that adaptive scaffolding was effective for increasing descriptive 
knowledge (e.g., a contrast of the procedural knowledge, also known as “know-how’”’), and 
increasing frequency of some SRL strategies. 


Further, Pintrich’s SRL Model is another model that emphasizes the role of motivation. He created 
the motivated strategies for learning questionnaire (MSLQ; Pintrich, Smith, Garcia, & Mckeachie, 
1993) and conducted empirical work on the relationship of SRL and motivation (Pintrich, Marx, & 
Boyle, 1993a). This model consists of four phases: a) forethought, b) monitoring, c) control, and 
d) reaction and reflection. Each of the phases has four different focal points for regulation: 
cognition, motivation/affect, behavior, and context. Therefore, the combination of phases and 
focal points provides a comprehensive picture that makes Pintrich’s SRL Model unique. 


Efklides’s model (2011) is another model that emphasizes metacognition and is known as the 
Metacognitive and Affective Model of Self-Regulated Learning. MASRL consisted of two levels: 
the person level (macro level) and the task-person level (micro level). This model contains most 


of the common factors within SR, such as cognition, motivation, metacognition, etc. In addition to 
the person level, the MASRL model contains a task-person level (micro level). At the micro level, 
four basic components exist, including cognition, metacognition, affect, and regulation of affect 
and effort. This level is bottom-up because students’ actions are controlled by their 


254 


metacognitive activities. This model illustrates the relationship between metacognition, 
motivation, and affect via two different levels in self-regulated learning. Also, Efklides (2002) 
created the Metacognitive Experiences Questionnaire, to examine the role of metacognitive 
experiences (e.g., feelings and judgments) in cognitive process and found that feeling and 
judgments serve as a monitor and controller in one’s decision-making process because a 
person’s perceptions are able to affect his/her ways of dealing with the task. 


Lastly, a model known as Socially Shared Regulated Learning model was proposed by Hadwin, 
Jarvela, and Miller (2011). One premise of this model is that group members need to establish a 
shared common ground and share their task perceptions and strategies to effectively collaborate 
with each other. In other words, learners’ regulatory actions are distributed because it involves 
an adaptive process of interaction with the group members (e.g., co-regulation; Hadwin, Oshige, 
Gress, & Winne, 2010). There are four loops within this model: a) groups negotiate and co- 
construct shared task perceptions, b) groups set shared goals and plans, c) groups strategically 
collaborate with each other through monitoring activities and changing their perceptions, goals, 
plans or strategies, and d) groups co-evaluate and co-regulate for future performance in order to 
build a collective level of regulation (Hadwin, Jarvela, & Miller, 2011). This concept of collective 
regulation sheds lights on the investigation of the shared motivation and emotion within SRL in 
the context of online learning because the diversity of online learners and the complexity of 
online resources seems to hinder learners’ effective collaborative learning to some extent. 


The Distinction Between SRL and Metacognition 


Due to a strong emphasis on metacognition in some models mentioned above, it is helpful for us 
to distinguish SRL and Metacognition to better understand the mechanisms of SRL and improve 
the effectiveness of SRL in practice. Metacognition is literally defined as a model of cognition at a 
meta-level (Nelson & Narens, 1994). At the simplest level, metacognition is thinking about your 
own thinking, which involves people’s awareness of the outcome of the monitoring process 
(Efklides, 2008). This might explain why metacognition and self-regulation are always 
intertwined; they are both related to self-awareness and regulatory action. 


Despite these shared core concepts, they are still different from each other due to the 
theoretical roots of the concepts (Fox & Riconscente, 2008). Specifically, the core meaning of 
metacognition focuses on the individual’s cognition. Yet, self-regulation is considered as the 
result of individual-environment interaction (Dinsmore, Alexander, & Loughlin, 2008). Secondly, 
there are three distinct facets of metacognition, namely, metacognitive knowledge (MK), 
metacognitive experiences (ME), and metacognitive skills (MS; Efklides, 2008). Specifically, MK 
refers to declarative knowledge such as language, memory, and so forth (Fabricius & 
Schwanenflugel, 1994). There are strong relations between language abilities and theory of mind 
(TOM), as well as with MK (Brown, Donelan-McCall, & Dunn, 1996; Lockl & Schneider, 2007). ME 
refers to metacognitive feelings and judgments the person has of task features (Efklides, 2001, 
2006), such as the feeling of familiarity, feelings of difficulty, or of confidence, and estimates of 
effort expenditure, etc. MS refers to procedural knowledge (e.g., know-how; Efklides, 2008). 


To activate MS, a person must be aware of the fluency of personal cognitive processing and be 
are that conflict or error has occurred in the current situation (Efklides, 2008). In other words, MS 
involves strategy use for planning, monitoring, executing, and evaluating task processing. 
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Therefore, SRL is a volitional action for achieving self-goals. To self-regulate, people need to be 
self-aware of the interaction between their goals and their cognition, emotions, behavior, and 
environment (Kuhl & Fuhrmann, 1998). Notably, ME is a pivot in that it makes people aware of 
their state of cognition and triggers corresponding control strategies to cope with the deficit 
between the current situation and the goals. 


The Applications of Self-regulated Learning in Practice 


There has been a substantial growth in the body of literature on self-regulated learning (SRL) in 
online learning (Lee, Watson & Watson, 2019; Littlejohn, Hood, Milligan, and Mustain, 2016; 
Broadbent and Poon, 2015; Broadbent, 2017; Bannert, Sonnenberg, Mengelkamp, & Pieger, 
2015; Miiller & Seufert, 2018; Lee, Lim, and Grabowski, 2010). 


M00Cs 


Lee, Watson, and Watson (2019) conducted a systematic literature review on self-regulated 
learning (SRL) in massive open online courses (MOOCs) by reviewing the studies on SRL in 
MOOCs published from 2008 to 2016 (47 articles). They first indicated that the topic of SRL in 
MOOCs has increasingly received attention from researchers and confirmed that SRL is one of 
the main themes of current and future research on MOOCs. During their review, they identified 
that there is a positive correlation between SRL and MOOC learning as well as MOOC learners 
and SRL strategies. 


Specifically, they concluded that there are three major motivational-related self-regulation 
strategies that positively affected MOOC learners’ learning outcomes including self-efficacy, 
task value, and goal setting. For example, Littlejohn and colleagues (2016) found that there is a 
distinct difference between the motivations of learners with high SRL scores and low SRI scores. 
The high SRL scores learners tended to be more engaged in the MOOCs by perceiving MOOCs 
as a professional learning opportunity to improve their level of knowledge and expertise rather 
than simply passing the exams. MOOCs, as non-formal learning, afford learners the freedom of 
learning based on their various needs. This study sheds light on how MOOCs should be 
designed and evaluated (Littlejohn et al., 2016). Instead of focusing on gaining a certificate or 
completing a course, a focus on encouraging learners to adopt more flexible self-regulated 
strategies, such as providing effective evaluation on their learning in relation to how well it 
related to their professional roles and how their learning should be adjusted towards their goals. 


In addition, Lee and colleagues (2019) also identified three behavioral and contextual regulation 
strategies including help seeking, time management, and effort regulation. In particular, the 
discussion forum is a vital source of help-seeking behaviors in a digital learning world (Milligan 
and Littlejohn, 2016). Also, a few articles proposed that time management is another critical 
factor influencing MOOC learning. For instance, poor time management skills have been shown 
to be a major cause of MOOC dropout (Nawrot and Doucet, 2014; Kizilcec and Halawa, 2015; 
Onah and Sinclair, 2016). These findings point to time management strategies as critical in the 
design or evaluation of MOOCs such as providing an adjustable timeline template for learners to 
customize their own learning schedules and offering timely feedback on feasibility. 


Lastly, Lee and colleagues (2019) presented several different SRL interventions and MOOC 
designs based on different theoretical frameworks of SRL that was adopted in the literature. 
However, the effect of SRL interventions is quite limited. Neither the self-regulated 


prompts/widget that allows MOOC learners to compare their behaviors with successful MOOC 
learners’ behaviors nor the study tips on SRL strategies could make any significant differences in 
improving the learning outcomes. In terms of MOOC designs, some design guidelines for 
facilitating SRL in MOOCs are providing design templates, promoting scaffolding activities, and 
supporting participants’ development of metacognitive strategies (Garcia, Tenorio, & Ramirez, 
2015; Park, Cha, & Lee, 2016; Milligan and Griffin, 2016). 


Learning Strategies 

As one of the most important learning strategies in the online learning environment, SRL 
strategies continue to receive educators’ and practitioners’ attention. Broadbent and Poon (2015) 
conducted a systematic review to examine the role of SRL strategies in academic achievement in 
online higher education settings from 12 studies that were published from 2004 to December 
2014. These SRL strategies are metacognition, time management, effort regulation, peer 
learning, elaboration, rehearsal, organization, critical thinking, and help seeking. However, they 
found out that the effect of SRL strategies in online learning is very limited. The authors believed 
that there are other factors that could be more important in online contexts other than SRL 
strategies. The authors further suggested that we should not assume that the application of SRL 
strategies in online learning is as effective as in the traditional classroom setting. We should not 
assume online learning will foster SRL strategies either. Instead, we should consider other 
factors, such as peer learning, since interaction with teachers is reduced in an online 
environment compared to in the classroom. Studies have shown that SRL strategies alone are 
insufficient to ensure academic success (Broadbent & Poon, 2015; Kizilcec & Halawa, 2015; 
Zheng, Rosson, Shih, & Carroll, 2015; Nawrot and Doucet, 2014). 


Broadbent (2017) later compared the role of online and blended learner's self-regulated 
learning strategies in their academic performance. She found out that peer learning and help 
seeking are the two most preferable strategies among online students. Time management and 
elaboration strategies are related to academic subject grades for both online and blended 
learning settings (Broadbent, 2017). 


Bannert and colleagues (2015) analyzed the short-term and long-term effects of students’ self- 
directed metacognitive prompts on navigation behavior and learning performance. They found 
that students who were provided with self-directed metacognitive prompts spent more time on 
the relevant web pages than those who do not have self-directed metacognitive prompts in use. 
This longer time spent on the relevant web pages contributed to a better transfer performance 
immediately after the learning session. However, there were no effects with respect to recall and 
comprehension (Bannert et al., 2015). 


Miiller and Seufert (2018) conducted a study on the effects of self-regulation prompts in 
hypermedia learning on learning performance and self-efficacy and found that prompted 
learners outperformed non-prompted learners only in the first performance test regarding the 
transfer. With respect to self-efficacy, learning with prompts may foster self-efficacy across 
learning sessions. For instructional purposes, the effect of self-regulation prompts in supporting 
deeper-level processing rather than in a more elaborated level of processing is the future 
direction. Also, the effect of prompts depends strongly on learner's compliance with them. The 
higher the level of compliance the students have, the more likely that they can transfer learning 
performance across different learning sessions (Miiller & Seufert, 2018). 


Lee et al. (2010) examined the effects of two scaffolding strategies - generative learning strategy 
prompts and metacognitive feedback - on learners’ comprehension and self-regulation in a 
computer-based learning environment (CBLEs). They found that learners' self-regulation serves 
as a mediator between the combination of generative learning strategy prompts with 
metacognitive feedback and the learners' comprehension. Their findings revealed that 
generative learning strategy prompts with metacognitive feedback significantly increased 
learners' self-regulation because metacognitive feedback required learners to monitor and 
refine their learning strategies during the learning. Providing generative learning strategy 
prompts with metacognitive feedback resulted in better recall and comprehension. This result 
might be because of the self-evaluation function that metacognitive feedback stimulates in 
learners. Metacognitive feedback led students to actively adjust their learning strategies so that 
they can adapt to new learning content. In short, an adaptive metacognitive feedback system is 
worthwhile in the future design of CBLEs (Lee et al., 2010) 


Self-regulation, metacognitive monitoring, and regulation 


Detinition and History 

Panadero (2017) analyzed and compared six models of Self-Regulated Learning (SRL). 
The first three models of SRL were developed by Zimmerman from a social-cognitive 
perspective. The first model, the Triadic Analysis of SRL, focuses on the interactions between the 
environment, the person, and behavior and is coordinated with Bandura’s model of social 
cognition. The second model shows the three cyclical phases of SRL including forethought, 
performance, and self-reflection. This model is the initial appearance of the interrelationship of 
metacognitive and motivational processes in SRL. The third model is a modification of the second 
model, in which several subprocesses were added into each cyclical phase of SRL. This last 
model became a representation of Zimmerman’s SRL model. This third model incorporates 
forethought, performance, and self-reflection (Panadero, 2017). The forethought phase 
represents students’ analysis and plans of tasks and goals and the performance phase is the 
execution of the task and plan. Students need to monitor their processes and motivate 
themselves throughout the phases and the self-reflection phase is when students assess their 
own performance of the task and make meaningful reasoning for their success or failure. 


The empirical studies in terms of the effect of this model showed that higher achievers showed 
more use of subprocesses from the cyclical model (DiBenedetto & Zimmerman, 2010); young 
male experts performed more SRL actions compared to young male novices in basketball 
(Cleary & Zimmerman, 2001). 


The next two models of SRL were developed by Boekaerts (1991) from the lens of goal setting in 
relation to emotion. One is named the six-component model of SRL, and another is the dual 
processing self-regulation model. Specifically, the six-component model of SRL is used mainly to 
train teachers and construct new instruments for research due to its well-defined structures 
(Panadero, 2017). The dual processing self-regulation model, as the representation of Boekaerts’ 
work, underscored the role of goal paths in students’ behavior change. In particular, the 
appraisals are crucial to determine students’ goal paths. For example, students were most likely 
to go for the well-being pathway to protect their beliefs (e.g., self-concept of ability) if the task 
triggered their negative emotions. However, if the task is coherent with their needs and goals, 


their positive emotions were triggered and this would lead to a growth pathway in which 
students would amplify their competence (Boekaerts & Cascallar, 2006; Boekaerts, 2011). 


The empirical support of the dual processing model indicates that competence and value 
appraisals positively influenced students’ outcome assessment and reported effort. For example, 
students who had reported that they were capable of doing their homework turn out to invest 
more effort in doing math homework because they produce more positive emotions during the 
task (Boekaerts & Voeten, 2003; Boekaerts & Rozendaal, 2007). Another major contribution that 
Boekaerts (1999) made is the development of the Online Motivation Questionnaire (OMQ), which 
measures the “sensitivity to learn in concrete situations.” For instance, on this questionnaire, 
students self-report their perceptions regarding a certain task (e.g., feeling about this task and 
the effort they will spend on this task). Then they report how they evaluate their feelings and 
attributions after the task (Boekaerts, 2002). 


A widely used model in the computer-supported learning settings (Panadero, Klug, & Jarvela, 
2015), was developed by Winne and Hadwin (1998) by integrating the metacognitive 
perspective. Notably, instead of emphasizing the role of emotion, such as exists in the Dual 
Processing Model, this model only mentions motivation. This model describes a feedback loop 
in which students cognitively monitor their activities and change strategies (Winne & Hadwin, 
1998). There are four phases within this model: a) task definitions (e.g., the understanding of a 
concrete task), b) goal setting and planning (e.g., goals and a plan to achieve the task), c) 
enacting study tactics (e.g., the use of strategies to attain the goals), and d) metacognitively 
adapting studying (e.g., a volitionally long-term and deep changes in motivations, beliefs and 
strategies for the future after the completion of the previous phases) 


Winne and colleagues have more recently explored the effect of this model in computer- 
supported learning environments. They found that tracing students’ online learning data had 
brought promising insights to the SRL field and provided a new approach to measure SRL 
(Winne, Hadwin, & Gress, 2010; Winne & Hadwin, 2013; Winne & Baker, 2013). For example, 
Azevedo and Hadwin (2005) found that adaptive scaffolding was effective for increasing 
descriptive knowledge (e.g., a contrast of the procedural knowledge, also known as “know- 
how”’), and increasing frequency of some SRI strategies. 


Further, Pintrich’s SRL Model is another model that emphasizes the role of motivation. He created 
the motivated strategies for learning questionnaire (MSLQ; Pintrich, Smith, Garcia, & Mckeachie, 
1993) and conducted empirical work on the relationship of SRL and motivation (Pintrich, Marx, & 
Boyle, 1993). This model consists of four phases: a) forethought, b) monitoring, c) control, and d) 


reaction and reflection. Each of the phases has four different focal points for regulation: 


cognition, motivation/affect, behavior, and context. Therefore, the combination of phases and 
focal points provides a comprehensive picture that makes Pintrich’s SRL Model unique. 


Efklides’s model (2011) is another model that emphasizes metacognition and is known as the 
Metacognitive and Affective Model of Self-Regulated Learning. MASRL consisted of two levels: 
the person level (macro level) and the task-person level (micro level). This model contains most 
of the common factors within SR, such as cognition, motivation, metacognition, etc. In addition to 
the person level, the MASRL model contains a task-person level (micro level). At the micro level, 
four basic components exist, including cognition, metacognition, affect, and regulation of affect 
and effort. This level is bottom-up because students’ actions are controlled by their 
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metacognitive activities. This model illustrates the relationship between metacognition, 
motivation, and affect via two different levels in self-regulated learning. Also, Efklides (2002) 
created the Metacognitive Experiences Questionnaire, to examine the role of metacognitive 
experiences (e.g., feelings and judgments) in cognitive process and found that feeling and 
judgments serve as a monitor and controller in one’s decision-making process because a 
person’s perceptions are able to affect his/her ways of dealing with the task. 


Lastly, a model known as Socially Shared Regulated Learning model was proposed by Hadwin, 
Jarvela, and Miller (2013). One premise of this model is that group members need to establish a 
shared common ground and share their task perceptions and strategies to effectively collaborate 
with each other. In other words, learners’ regulatory actions are distributed because it involves 
an adaptive process of interaction with the group members (e.g., co-regulation; Hadwin, Oshige, 
Gress, & Winne, 2010). There are four loops within this model: a) groups negotiate and co- 
construct shared task perceptions, b) groups set shared goals and plans, c) groups strategically 
collaborate with each other through monitoring activities and changing their perceptions, goals, 
plans or strategies, and d) groups co-evaluate and co-regulate for future performance in order to 
build a collective level of regulation (Hadwin, Jarvela, & Miller, 2011). This concept of collective 
regulation sheds lights on the investigation of the shared motivation and emotion within SRL in 
the context of online learning because the diversity of online learners and the complexity of 
online resources seems to hinder learners’ effective collaborative learning to some extent. 


The Distinction Between SRL and Metacognition 

Due to a strong emphasis on metacognition in some models mentioned above, it is helpful for us 
to distinguish SRL and Metacognition to better understand the mechanisms of SRL and improve 
the effectiveness of SRL in practice. Metacognition is literally defined as a model of cognition at a 
meta-level (Nelson & Narens, 1994). At the simplest level, metacognition is thinking about your 
own thinking, which involves people’s awareness of the outcome of the monitoring process 
(Efklides, 2008). This might explain why metacognition and self-regulation are always 
intertwined; they are both related to self-awareness and regulatory action. 


Despite these shared core concepts, they are still different from each other due to the 
theoretical roots of the concepts (Fox & Riconscente, 2008). Specifically, the core meaning of 
metacognition focuses on the individual’s cognition. Yet, self-regulation is considered as the 
result of individual-environment interaction (Dinsmore, Alexander, & Loughlin, 2008). Secondly, 
there are three distinct facets of metacognition, namely, metacognitive knowledge (MK), 
metacognitive experiences (ME), and metacognitive skills (MS; Efklides, 2008). Specifically, MK 
refers to declarative knowledge such as language, memory, and so forth (Fabricius & 
Schwanenflugel, 1994). There are strong relations between language abilities and theory of mind 
(TOM), as well as with MK (Brown, Donelan-McCall, & Dunn, 1996; Lockl & Schneider, 2007). ME 
refers to metacognitive feelings and judgments the person has of task features (Efklides, 2001, 
2006).), such as the feeling of familiarity, feelings of difficulty, or of confidence, and estimates of 
effort expenditure, etc. MS refers to procedural knowledge (e.g., know-how; Efklides, 2008). 


To activate MS, a person must be aware of the fluency of personal cognitive processing and be 
are that conflict or error has occurred in the current situation (Efklides, 2008). In other words, MS 


involves strategy use for planning, monitoring, executing, and evaluating task processing. 
Therefore, SRL is a volitional action for achieving self-goals. To self-regulate, people need to be 
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self-aware of the interaction between their goals and their cognition, emotions, behavior, and 
environment (Kuhl & Fuhrmann,1998). Notably, ME is a pivot in that it makes people aware of 
their state of cognition and triggers corresponding control strategies to cope with the deficit 
between the current situation and the goals. 


The Applications of Self-regulated Learning in Practice 
There has been a substantial growth in the body of literature on self-regulated learning (SRL) in 
online learning (Lee, Watson & Watson, 2019; Littlejohn, Hood, Milligan, and Mustain, 2016; 
Broadbent and Poon, 2015; Broadbent, 2017; Bannert, Sonnenberg, Mengelkamp & Pieger, 2015; 
Mueller and Seufert, 2018; Lee, Lim, and Grabowski, 2010). 


M00Cs 


Lee, Watson, and Watson (2019) conducted a systematic literature review on self-regulated 
learning (SRL) in massive open online courses (MOOCs) by reviewing the studies on SRL in 
MOOCs published from 2008 to 2016 (47 articles). They first indicated that the topic of SRL in 
MOOCs has increasingly received attention from researchers and confirmed that SRL is one of 
the main themes of current and future research on MOOCs. During their review, they identified 
that there is a positive correlation between SRL and MOOC learning as well as MOOC learners 
and SRL strategies. 


Specifically, they concluded that there are three major motivational-related self-regulation 
strategies that positively affected MOOC learners’ learning outcomes including self-efficacy, 
task value, and goal setting. For example, Littlejohn, Hood, Milligan, and Mustain (2016) found 
that there is a distinct difference between the motivations of learners with high SRL scores and 
low SRL scores. The high SRL scores learners tended to be more engaged in the MOOCs by 
perceiving MOOCs as a professional learning opportunity to improve their level of knowledge 
and expertise rather than simply passing the exams. MOOCs, as non-formal learning, afford 
learners the freedom of learning based on their various needs. This study sheds light on how 
MOOCs should be designed and evaluated (Littlejohn, 2016). Instead of focusing on gaining a 
certificate or completing a course, a focus on encouraging learners to adopt more flexible self- 
regulated strategies, such as providing effective evaluation on their learning in relation to how 
well it related to their professional roles and how their learning should be adjusted towards their 
goals. 


In addition, Lee and colleagues (2019) also identified three behavioral and contextual regulation 
strategies including help seeking, time management, and effort regulation. In particular, the 
discussion forum is a vital source of help-seeking behaviors in a digital learning world (Milligan 
and Littlejohn, 2016). Also, a few articles proposed that time management is another critical 
factor influencing MOOC learning. For instance, poor time management skills have been shown 
to be a major cause of MOOC dropout (Nawrot and Doucet, 2014; Kizilcec and Halawa, 2015; 
Onah and Sinclair, 2016). These findings point to time management strategies as critical in the 
design or evaluation of MOOCs such as providing an adjustable timeline template for learners to 
customize their own learning schedules and offering timely feedback on feasibility. 


Lastly, Lee and colleagues (2019) presented several different SRL interventions and MOOC 
designs based on different theoretical frameworks of SRL that was adopted in the literature. 
However, the effect of SRL interventions is quite limited. Neither the self-regulated 


prompts/widget that allows MOOC learners to compare their behaviors with successful MOOC 
learners’ behaviors nor the study tips on SRL strategies could make any significant differences in 
improving the learning outcomes. In terms of MOOC designs, some design guidelines for 
facilitating SRL in MOOCs are providing design templates, promoting scaffolding activities, and 
supporting participants’ development of metacognitive strategies (Garcia Espinosa, Tenorio 
Septlveda, and Ramirez Montoya, 2015; Park, Cha, and Lee, 2016; Milligan and Griffin, 2016). 


Learning Strategies 

As one of the most important learning strategies in the online learning environment, SRL 
strategies continue to receive educators’ and practitioners’ attention. Broadbent and Poon (2015) 
conducted a systematic review to examine the role of SRL strategies in academic achievement in 
online higher education settings from 12 studies that were published from 2004 to December 
2014. These SRL strategies are metacognition, time management, effort regulation, peer 
learning, elaboration, rehearsal, organization, critical thinking, and help seeking. However, they 
found out that the effect of SRL strategies in online learning is very limited. The authors believed 
that there are other factors that could be more important in online contexts other than SRL 
strategies. The authors further suggested that we should not assume that the application of SRL 
strategies in online learning is as effective as in the traditional classroom setting. We should not 
assume online learning will foster SRL strategies either. Instead, we should consider other 
factors, such as peer learning, since interaction with teachers is reduced in an online 
environment compared to in the classroom. Studies have shown that SRL strategies alone are 
insufficient to ensure academic success (Broadbent & Poon, 2015; Kizilcec & Halawa, 2015; Zheng 
et al., 2015; Nawrot and Doucet, 2014). 


Broadbent (2017) later compared the role of online and blended learner's self-regulated 
learning strategies in their academic performance. She found out that peer learning and help 
seeking are the two most preferable strategies among online students. Time management and 
elaboration strategies are related to academic subject grades for both online and blended 
learning settings. 


Bannert, Sonnenberg, Mengelkamp, and Pieger (2015) analyzed the short-term and long-term 
effects of students’ self-directed metacognitive prompts on navigation behavior and learning 
performance. They found that students who were provided with self-directed metacognitive 
prompts spent more time on the relevant web pages than those who do not have self-directed 
metacognitive prompts in use. This longer time spent on the relevant web pages contributed toa 
better transfer performance immediately after the learning session. However, there were no 
effects with respect to recall and comprehension. 


Mueller and Seufert (2018) conducted a study on the effects of self-regulation prompts in 
hypermedia learning on learning performance and self-efficacy and found that prompted 
learners outperformed non-prompted learners only in the first performance test regarding the 
transfer. With respect to self-efficacy, learning with prompts may foster self-efficacy across 
learning sessions. For instructional purposes, the effect of self-regulation prompts in supporting 
deeper-level processing rather than in a more elaborated level of processing is the future 
direction. Also, the effect of prompts depends strongly on learner's compliance with them. The 
higher the level of compliance the students have, the more likely that they can transfer learning 
performance across different learning sessions. 


Lee, Lim, and Grabowski (2010) examined the effects of two scaffolding strategies - generative 
learning strategy prompts and metacognitive feedback - on learners’ comprehension and self- 
regulation in a computer-based learning environment (CBLEs). They found that learners' self- 
regulation serves as a mediator between the combination of generative learning strategy 
prompts with metacognitive feedback and the learners' comprehension. Their findings revealed 
that generative learning strategy prompts with metacognitive feedback significantly increased 
learners' self-regulation because metacognitive feedback required learners to monitor and 
refine their learning strategies during the learning. Providing generative learning strategy 
prompts with metacognitive feedback resulted in better recall and comprehension. This result 
might be because of the self-evaluation function that metacognitive feedback stimulates in 
learners. Metacognitive feedback led students to actively adjust their learning strategies so that 
they can adapt to new learning content. In short, an adaptive metacognitive feedback system is 
worthwhile in the future design of CBLEs. 


Guided instruction vs inquiry-based instruction: the need for scaffolding. 

The debate on whether education is best delivered through direct and purposeful guidance or 
through student-driven knowledge construction has been raging for over 50 years in educational 
and psychological circles (Ausubel, 1964; Craig, 1956; Hmelo-Silver, Duncan, & Chinn, 2007; 
Kirschner, Sweller, & Clark, 2006; Mayer, 2004; Schmidt, 2000; Schmidt, Loyens, van Gog, & 
Paas, 2007; Sweller, Kirschner, & Clark, 2007). The proponents of minimally-guided instruction 
(MGI), also known as discovery learning (DL), experiential learning (EL), problem-based 
learning (PBL), inquiry learning (IL), and constructivist learning (CL) insist that learners learn 
best when they must uncover, discover, and construct essential information for themselves 
(Kirschner et al., 2006). In the MGI landscape, students operate as solo or group sense-makers 
who are active in building coherent and organized knowledge structures (Mayer, 2004). 
Branson, Brown, and Cocking (2000) cautioned that, in this minimally-guided approach, 
instructors should not confuse a theory of pedagogy (teaching by some form of minimal 
guidance) with a theory of knowing (constructivism), but that, rather than only telling, teachers 
do need to incorporate students’ prior knowledge and beliefs into the instructional schema. 


Some proponents claim that problem-based and/or inquiry learning are not minimally guided at 
all (Hmelo-Silver et al., 2007; Schmidt et al., 2007). On this point, proponents posit that PBL 
operates with guidance that is both flexible and respectful of human cognitive architecture 
(Schmidt et al., 2007). Further, these proponents say that both PBL and IL are highly scaffolded 
learning approaches which move learners from simple concepts to more complex queries, 
which offers direction in learning, while ushering the students into a more independent learning 
mode (Hmelo-Silver et al., 2007; Schmidt et al., 2007). This idea of helping learners become 
independent rings true considering the quest to promote active learning (Branson et al., 2000). 


Active learning employs metacognition to produce sense-making, self-assessment, and 


reflection (Branson et al., 2000). In turn, prior knowledge dictates how learners will organize and 
interpret their learning environment and how they will reason, solve problems, remember, and 
acquire their new knowledge (Branson et al., 2000). Therefore, teaching for understanding 
needs to include self-efficacy and metacognition to be effective from the students’ perspective 
(Branson et al., 2000). Hmelo-Silver et al. (2007) claimed that, contrary to the opinion of MGI 
skeptics, PBL and IL can encourage deep and meaningful learning and gains in student 
achievement on standardized tests because these strategies do make use of scaffolding and 


guidance as students pursue sense-making. Some proponents of MGI elaborate that, perhaps 
knowledge should not be taught at all, but that instruction should be aimed toward helping 
students learn the skills of acquiring knowledge (Kuhn, 2007). 


The skeptics of MGI point to less constructivist notions or methods that discriminate novices from 
experts and insist that the cognitive architecture of the novice learner is such that these learners 
require direct instruction and guidance on procedures and concepts within a discipline 
(Kirschner et al., 2006; Sweller, 2007). The guided instruction (GI) model is more teacher-centric 
and uses teacher explanation, modeling, and feedback to teach concepts and skills with student 
progress monitoring throughout the instructional process (Yeh, 2009). For direct instruction to 
be effective, instructors should identify the learning intentions, criteria for student success, 
methods of engagement in the lesson, instructional design, opportunities for guided practice, 
lesson closure, and methods of independent practice (Hattie, 2009). The skeptics of MGI also 
point out that certain knowledge must be taught (biologically secondary knowledge) and that, to 
that end, cognitive architecture must rule the methodology of teaching (Sweller, 2007). These 
skeptics point to the foundational principles of cognitive architecture, the research that supports 
guided learning (GL), and the role of guidance in the transition from novice to expert (Kirschner, 
2006). Despite the common parlance that guided instruction is outdated, meta-analyses point to 
high effect sizes for regular ability students (d = 0.99), lower ability students (d = 0.86), and 
high-level comprehension (d = 0.54) (Hattie, 2009). 


In comparing and contrasting the MGI and GI models, Mayer (2004) stated that, in regard to 
transfer of knowledge, research demonstrates that methods supportive of cognitive activity, 
instructional guidance, and curricular focus are superior to those depending on behavioral 
activity, pure discovery, and unstructured exploration. Guided instruction has recently been 
shown to advantageous in both learning outcomes and self-efficacy in elementary school science 
learning (Hushman & Marley, 2015). On the opposite end of the education spectrum, researchers 
found that medical students preferred guided case-based instruction to unguided formats 
(Adamas-Rappaport et al., 2013). Ina study of German medical students, researchers found that 
self-directed learners perceived they had higher knowledge level than conventionally taught 
students (Peine, Kabino & Sprecklesen, 2016). 


Deciphering the best practices of MGI versus GI is fraught with conflicting studies. Below is a 
discussion of some of the most common forms of MGI that have emerged as beneficial to 
learning and student satisfaction (PBL and IL), as well as a description of two important reasons 


that MGI could be used effectively, namely, scaffolding and feedback. 


Problem-based learning 

Problem-based learning, also known as scenario- or case-based learning, is a method of 
instruction in which learners practice thinking skills in true-to-life situations (Clark & Mayer, 
2016). The practice may take place orally or in written form. It was first used in medical 
curriculum by McMasters University in Canada. Criticism of PBL includes the lack of evidence for 
improved academic achievement and that PBL is less efficient than instruction that provides 
greater guidance in the learning process (Colliver, 2000; Renkl, 2011; Schmidt, 2010). According 
to Renkl (2011), this is due to two factors. 


First, students in PBL may not have accurately encoded for general rules or principles of a 
discipline such that their investigations into one problem are transferred to new situations. 
Secondly, learners may not have noticed the relevant information that would lead to deep 
learning. In assessment measures after PBL, this method of instruction had the greatest effect on 
the knowledge of concepts that are associated and link principles (Gijbels, Dochy, Van den 
Bossche, & Segers, 2005). Further, problem-based learning emphasizes meaning and 
understanding rather than knowledge acquisition (Hattie, 2009). Sweller et al. (2007) explained 
that PBL/IL cannot work effectively due to the violation of the human cognitive architecture that is 
inherent in its premise; that is, if human cognitive architecture is such that working memory is 
severely hampered when interacting with new information, the worst thing novice learners could 
do is to engage in problem solving in unfamiliar mental territory. Problem solving searches 
impose a heavy extraneous cognitive load and Sweller et al. (2007) suggested that there would 
be no benefit to withholding information that could be helpful in learning a discipline. 
Furthermore, the PBL may be contrary to the worked example effect, which is supposedly 
supportive of learning and skill acquisition by providing problems and solutions (Renkl, 2014). 


The worked example effect is strongest when the examples are given at the earliest stages of 
cognitive skill acquisition (Renkl, 2014). If, as Chi, Bassok, Lewis, Reimann, and Glaser (1989) 
stated, worked examples are only effective if students encode correctly (read for understanding, 
monitor themselves for comprehension, or use the examples as a reference instead of as a 
pattern for solutions), it may prove difficult for some students to effectively use the self-searching 
approach implicit in PBL. It seems undeniable that superior learning outcomes are a result of 
using worked examples when students use correct encoding strategies (Chi et al., 1989; Renkl, 
2011). Despite the criticisms, PBL has been implemented in K-12 up through post-graduate 
medical education (Schmidt, 2010). 


Medical education is one of the largest use-cases of PBL (Colliver, 2000; Schmidt, 2010). After 
compiling research reviews, Colliver (2000) concluded that there was no overtly convincing 
evidence of the utility of PBL in medicine to improve the knowledge foundation of the medical 


students, nor their clinical performance as gauged by large effect sizes. Schmidt (2009, 2010) 
disagreed, citing that, in reviewing large numbers of studies, PBL in medical education is 
effective in multiple ways, such as improving test scores in diagnostic reasoning, professional 
competencies, and interpersonal interactions. Further, Schmidt (2010) cited that the perception 
of PBL in medical training is that it is superior in relation to the quality of education, drop-out 
rates, and acquired medical knowledge. 


PBL/IL environments depend on scaffolding to achieve progress through the learning construct 
(Hmelo-Silver et al., 2007). Scaffolding is cited as the main form of guidance in these learning 


265 


environments. This involves learners working within complex tasks to devolve problems that 
would be beyond the learners’ capabilities to solve and bringing those problems into 
manageable pieces within the students’ individual zone of proximal development (Hmelo-Silver 
et al., 2007; Fernandez, Wegerif, Mercer, Rojas-Drummond, 2001; Wood, Bruner, & Rojas- 
Drummond, 1976). 


Inquiry Learning 

Inquiry learning is collaboration-based, and the methods involve students learning content and 
reasoning skills that are specific to a discipline (Hmelo-Silver et al., 2007). IL is viewed by some 
as being indistinguishable from PBL, except in its origins (PBL has its origins in medical 
education and IL has its origins in scientific inquiry) (Hmelo-Silver et al., 2007). In a review of 
four meta-analyses and over 200 studies, Hattie (2009) reported an average effect size of d = 0.35 
for inquiry-based learning. Inquiry based instruction has been effective in atypical populations 
in which student thinking and learning was undervalued and has been shown to produce transfer 
of critical thinking skills and attitudes toward instruction (Hattie, 2009). 


Scaffolding 


Scaffolding is a learning strategy that uses student engagement as a tool to build autonomy in 
learners (Davis & Linn, 2000). Scaffolding can be implemented through reflection or through 
more directed learner support (Davis & Linn, 2000; Kim & Lim, 2019). Traditionally, scaffolding 
has implied the interaction between the instructor and the student that places the teacher ina 
supporting role in the learning process by helping students tackle problems that they would not 
be able to solve with their current abilities (Puntambeker & Hubscher, 2005; Wood et al., 1976) 
Increasingly, scaffolding has not only encompassed human input but also computer software, 
resources, and curricula used to aid students’ learning (Puntambeker & Hubscher, 2005). 
Scaffolding is valuable to learners by providing them with an overview of the expected task and 
delineating the scope of the task (Fernandez et al., 2001). Furthermore, scaffolding assists 
students by minimizing frustration because it helps them understand the course requirements 
and how to accomplish them (Fernandez et al., 2001). It also serves to help students reduce the 
number of steps required to complete the assignment and provides an ideal representation of 
the task for students to emulate (Fernandez et al., 2001). 


Successful scaffolding methods include intentionally designed prompts that help students 
integrate knowledge and monitor their own progress (Davis & Linn, 2000). Prompts are useful as 
precursors to feedback (Hattie, 2012). Prompts that direct students to consider the organization 
of the lesson, to elaborate on examples, to monitor their understanding, and to remediate when 
comprehension is lacking show the most successful learning because they support the cognitive, 
metacognitive, and self-monitoring aspects of learner self-regulation (Nuckles, Hubner, & Renkl, 
2009). Supportive scaffolding encompasses providing domain-specific knowledge or guidance 
to learners on what aspects of the problem they should consider in their strategy (Kim & Lim, 
2019). It can be presented as an explanation, visual aid, template, or reference (Kim & Lim, 
2019). 


Kim and Lim have shown that reflective scaffolding employs metacognitive questions that direct 
learners to explain their own metacognitive processes and behaviors. Reflective scaffolding can 
take the form of process maps, models depicting expert solutions, and exploratory questions or 
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hints. Recent studies have shown that reflective scaffolding is more effective for building 
cognitive and social presence in learners than is supportive scaffolding. Additionally, reflective 
scaffolding improved students’ problem representation, and achievement compared to 
supportive scaffolding (Kim & Lim, 2019). 


In PBL and IL, scaffolding can serve to decrease cognitive load on the learner because the 
scaffolding supports the learners’ inquiry process through assisting with sense-making, process 
management, discussion, explanation, and reflection (Hmelo-Silver, 2007). The act of articulating 
their ideas throughout the learning helps students identify their own knowledge gaps and 
misconceptions, which can help them become more autonomous learners (Davis & Linn, 2000). 


Feedback 

Feedback can promote accurate self-evaluation through modeling good performance (Adcroft, 
2011: Miller, 2009). Feedback can also be an impetus for change (Alharbi, 2017) as long as the 
substance of the feedback is usable by students such that it closes the chasm between a model 
performance and their own performance and directly affects their future work and 
understanding (Glover & Brown, 2006; Walker, 2009). Glover and Brown (2006) labeled this type 
of student-directed evaluation as feedback coupled with feed-forward. In online instruction, 
feedback is most effective when it combines an explanation of the task the learner is to perform 
with instructions on how to complete the task (Clark & Mayer, 2016). Hattie (2009), in a synthesis 
of over 800 meta-analyses, states that of over 100 factors that influence achievement, feedback 
was ranked as number 10. Krause and Stark (2010) found that, not only did problem-based and 
example-based environments promote learning but that reflection prompts and feedback 
improved learning. Further, individuals benefit from individual feedback more than group 
feedback. Feedback can be especially challenging in online environments due to the distance 
between instructors and learners and due to the nature of online learning platforms which can 
sometimes be restrictive (Alharbi, 2017). 


To be most effective, feedback must possess certain qualities (see Table 1) (Coll, Rochera, 
Gispert, & Diaz-Barriga, 2013; Gibbs & Simpson, 2005; Glover & Brown, 2006; Harvey, Radomski, 
& O’Connor, 2013). Adcroft (2011) found that there is a dissonance between instructors and 
students regarding feedback that is a result of an expectation gap. Generally, students and 
instructors agree that feedback is important to clarify class expectations and to make class 
performance standards clear; however, academics place greater value on feedback as a tool for 
learning, are more inclined to see feedback as an indicator of prior knowledge or understanding 
gaps, and to feel that students see feedback as important compared to students (Adcroft, 2011). 
Conversely, students are stronger in their belief that feedback improves their performance than 
are academics (Adcroft, 2011). Mutch (2003) raised the admonition that instructors and students 
need to view the idea of feedback as developmental and, as such, should expect feedback to be 
dependent on the student’s capacity to understand and apply the feedback. This developmental 
approach would increase the value of the exercise as students learn from and apply the 
evaluation (Mutch, 2003). Adcroft (2011) asserted that students cannot learn from feedback if 
they do not value it or recognize how to apply it to improve academic performance. 
Furthermore, Adcroft (2011) added that students who are only focused on performance 
assessment feedback cannot receive the benefits that come from other types of feedback and 


that this further complicates the instructors’ frustration at what they view as students wasting the 
opportunity to improve. 


Kamp et al (2014), in a study of peer feedback, stated that peer feedback has the potential to 
improve individual performance within the collaborative context if it is accompanied by 
reflection and goal setting activities. Papinczak, Young, and Groves (2007) found that peer 
feedback can induce a greater sense of responsibility in individuals toward the group. 


Feedback Characteristic 


Task Appropriate 


Applied to Future Learning 


Hattie (2009; 2012) related that the most crucial form of feedback necessary to boost student 
improvement is that of student to teacher. Using student input, whether by assessment or by 
student activity progress and understanding, allows instructors to adjust their teaching so that 
they can correct the students’ misconceptions in learning. 


Zone of Proximal Development 

The zone of proximal development (ZPD) is the continuum between what learners can 
accomplish on their own, what they can accomplish with help, and what they are unable to 
accomplish (Knestrick, 2012). The term was introduced by Lev Vygotsky, a psychologist, in the 
1930’s to help instructors to understand where an individual student is most primed to learn. 
Building on what students already know to help them advance in learning (scaffolding) is 
combined with feedback through formative assessments to keep students in the best target area 
for progression (Knestrick, 2012). Part of tapping into the ZPD is to understand the 
connectedness of language and discourse to the process of learning and concept development 
(Fernandez et al., 2001). In learning environments, Borthick, Jones, and Wakai (2003) warned 


that designs that encourage the intermingling of the cognitive and social aspects of learning 
should be thoughtfully carried out so as not to subjugate one aspect over the other. Online 
delivery must maintain the best practices in providing learners with relevant objectives, just-in- 
time learning opportunities, task configuration, facilitation of activities, and keeping the course 
fluid in terms of recapturing students’ zones of proximal development as learning happens 
(Borthick et al., 2003). 


Learning Approaches and MO00Cs 


The way a learner approaches the subject at hand can have a large impact on their learning 
outcome when it comes to the use of MOOCs. While the IL and PBL have been said to be 
indistinguishable in everything but their origins (Hmelo-Silver et al., 2007), the implementation 
of IL has had much more success in the MOOC environment than PBL has. The draw that the 
MOOC model has on the learner with IL is so interconnected, that it has been suggested that the 
IL resolution be considered a factor of the MOOC effectiveness analysis (Kovanovic¢ et al., 2018). 
Meanwhile, learning engineers have found it difficult to upscale PBL activities into MOOCs 
(Spoelstra, van Rosmalen, Houtmans, & Sloep, 2015). Currently this is a limitation regarding the 
lack of face-to-face interaction with other MOOC users, and that traditional PBL has been 
composed of team-based tasks (Jablokow, Matson, & Velegol, 2014). While the upscaling of 
traditional means of PBL in MOOCs is behind compared to its IL counterpart, this does not mean 
PBL has no place in MOOCS; instead, this means that alternative means of PBL must be 
investigated. 


Problem-Based Learning and Blended Learning 

Problem-based learning is frequently used in courses that are project oriented, shown by 
Tambouris, Zotou, and Tarabanis (2014). The steps of problem solving become the focus as much 
as the knowledge acquisition. Students may struggle with a blended PBL environment due to 
needing to learn many new tools and techniques. Also, instructors may have an increased 
workload when initiating a blended PBL since the PBL model and redesign of a course for 
blended delivery require rethinking of materials, assessments, and content to fit the structure of 
the course. Additional burden is placed on the instructor to modify the knowledge that students 
are creating to ensure that it meets the requirements of the course and is scientifically accurate 
(Tambouris et al., 2014). 


Health professions learning is one area that is dominated by PBL (Woltering, Herrler, Spitzer, & 
Sprekelsen, 2009). Woltering and colleagues (2009) found that students participating ina 
blended PBL environment experienced improvement in satisfaction, motivation, and subjective 
learning gains compared to students in a traditional PBL environment. No differences between 
the groups were found by test results and the tutors’ opinions of the two environments showed 
no differences. De Jong, Krumeich, and Verstegen (2017) investigated blended PBL in three 
health master’s programs and found that, depending upon the way the blending is configured, 
all principles of PBL (constructive, collaborative, self-directed, and contextual) may not be 
evident. Important principles for blended PBL were offered by de Jong et al. (2017), namely that 
face-to-face meetings should be designed to acquaint students with each other, online rules must 
be agreed upon by students, and visibility of the instructor and other students must be 
maintained and fostered through feedback. 


Munezero and Bekuta (2016) found that forestry students at a Kenyan university produced 
solutions that were more contextually relevant and helped students with interpersonal skills, 
interdisciplinary skills, and technical skills that would benefit them on the job compared to 
curriculum alone. 


Computer-mediated collaborative learning 
Overview 


Computer-Supported Collaborative Learning (CSCL) refers to collaborative learning that is 
facilitated or mediated by computers and networked devices in a synchronous (e.g., a chat 
room) or asynchronous (e.g., a discussion forum) manner (Sawyer, 2005). CSCL research centers 
around the challenge of combining computer support and collaborative learning to enhance 
learning outcomes. Researchers believe that the analytic nature of situated practices and 
interactional processes position CSCL as a valuable combination in online learning 
environments (Sawyer, 2005). 


The Historical Evolution of Computer-Supported Collaborative Learning (CSCL) 

The field of CSCL started with three small-scale school programs including the ENFI Project at 
Gallaudet University, the Computer Supported Intentional Learning Environment (CSILE) Project 
at the University of Toronto, and the Fifth Dimension (5thD) Project at the University of California, 
San Diego (Stahl, Koschmann, & Daniel, 2005). The three projects utilized computer-aided 
applications to improve students’ literacy skills. The field of CSCL experienced two early 
developmental stages. The first, in the 1960s, was advanced by behaviorists who recommended 
the use of computer-assisted instruction to aid students in memorizing facts via computerized 
drill and practice. The second stage, in the 1970s, was influenced by the cognitivists, who 
rejected a definition of learning as fact acquisition and retention and, instead, heralded the role 
of intelligent tutoring systems that could recognize student learning patterns and provide 
personalized feedback to students. Currently, CSCL draws on social constructivist and dialogical 
theories and relies on emerging educational technologies (Stahl et al., 2005). 


Consequently, the focus of CSCL has shifted from only providing instruction to enabling 
communication and scaffolding for students’ meaningful interaction (Stahl et al., 2005; 
Dillenbourg, Jarvela & Fischer, 2009). From the participants’ perspective, the evolution of the 
field of CSCL has shifted from a focus on individual learning to an analytic tool that is socially 
constructed, and group focused (Dillenbourg, Baker, Blaye & O’Malley, 1996; Stahl et al., 2005). 
Examining the group, instead of the individual, as the unit of analysis elevates CSCL into a set of 
process-oriented activities through which learning happens via constructing shared meaning 
among learners. 


The Analysis of Collaboration 
Collaborative learning lies at the heart of CSCL. Koschmann (2002) described CSCL as being 
concerned with meaning and meaning making in group settings and with the construction of 


designed artifacts through which learning occurs. The emphasis of meaning-making in the 
context of joint activity reveals the importance of collaboration. 


There are three distinct approaches used to analyze collaboration, namely, collaboration for 
distal (farther removed) outcomes, collaboration for proximal (nearer to the present) outcomes, 
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and collaboration as a form of learning in itself (Enyedy & Stevens, 2005). Different nuances of 
discussion can lead to different learning outcomes that are either proximal or distal in their 
impact. The most significant difference between distal and proximal collaboration is the level of 
learning outcomes. As shown in Table 1, collaboration for distal outcomes is associated with 
individual learning outcomes, whereas collaboration for proximal outcomes is related to 
collective learning outcomes. Distal collaboration requires identifying a collaborative discourse 
pattern and relating it to a distal outcome typically reified as a product (Erickson, 1986). Some 
examples are a fixed three-turn dialog sequence. One specific instance of this dialog sequence 
is the IRE (initiate-respond-evaluate), in which the teacher asks a question for which the answer 
is known, thereby initiating the dialog sequence. The student responds to the question with a 
short answer (respond), which the teacher immediately assesses as right or wrong (evaluate) 
categories of talk (e.g., accountable talk) or classroom-level structures (e.g., norms). These 
collaborative phenomena produce individual learning artifacts, signifying outcomes (e.g., test 
scores). Thus, distal collaboration is an effect-oriented approach to promote personalized 
learning outcomes. 


Enyedy and Stevens (2005) asserted that proximal collaboration underscores how collaborative 
interaction contributes to problem-solving. Because proximal collaboration is related to joint 
activity or collective processes, proximal outcomes in the conversations (e.g., intersubjectivity) 
relate to individual distal outcomes, such as better personal understanding of math concepts 
(Enyedy & Stevens, 2005). An excellent example is the Jeffersonian transcription conventions of 
conversation analysis (Atkinson & Heritage, 1984), which is a method that allows analysts to track 
how interactions unfold across participants by capturing a myriad of interaction details. 


Lastly, collaboration as a form of learning does not negate the contributions and dispositions of 
individual participants but shows distributed units of analysis that stretch across multiple 
participants and tools (Hutchins, 1995; Stevens, 2000). These group members interact ina 
“socio-cultural system” to create the outcomes of the group’s projects and influence the group’s 
performance (Hutchins, 1995; Stevens, 2000). Conversation Analysis (CA) and 
Ethnomethodology (EM) are two primary methods that align well with distributed collaboration 
because the analysis of the partnership will show how members themselves are jointly managing 
and maintaining cooperation. Indeed, the contrast between the traditional classroom setting for 
collaborative learning and collaborative learning in the online environment is striking. 


Table 3. Three approaches to studying collaboration 


The three distinct approaches to 


. Learning Outcomes Examples 
analyze collaboration 


; ‘ ja IRE; Revoicing; Exploratory talk; Accountable talk; 
Collaboration-for-distal-outcomes Individual exe a Y : 
Knowledge building discourse; Norms 
Transcription conventions 
of conversation analysis; 
“propositional content” of talk 


Collective (to explain 


Collaboration-for-proximal-outcomes cies 
individual) 


Collaboration as a form of learning in Collective Conversation Analysis (CA) 
itself and Ethnomethodology (EM) 


Argumentation skills in CSCL 

As one of the features of collaborative learning, argumentation offers opportunities for students 
to engage in collaborative discourse so that their conceptual understanding and skills and 
capabilities with scientific reasoning can be enhanced (Andriessen & Baker, 2005). Arguing to 
learn can be intimidating for students and treated as an interference in the learning process. 
Distinctions must be made between collaborative argumentation and oppositional 
argumentation to understand the role of argumentation in learning. 


Educational practitioners need to make arguing in learning environments less oppositional in 
nature; therefore, instructors should consider five principles of collaborative argumentation 
(Andriessen & Baker, 2005). These principles are: 


a) Change in view. As a result of argumentation, viewpoints become refined and more 
nuanced. Opponents should remain open-minded about the discrepancy in their 
positions and maintain a degree of empathy learners holding contrary views to their own. 
A change in perspective will help learners understand and consider the beliefs of others 
in a respectful manner. 


Making knowledge explicit. Learners reach a deep level of learning by explaining the 
thinking and reasoning behind their behavior choices. For example, when learning 
something privately, a person has a chance to think as deeply, but cannot expand the 
breadth of multiple perspectives alone. To cultivate different perspectives requires self- 
critique, which allows learners to investigate alternate opinions. 


Conceptual change. The learners’ arguments may raise doubt about initial opinions by 
distinguishing similar concepts and elaborating new definitions of them. For instance, in 
a knowledge-building community, every idea is treated as potentially improvable. 
Argumentation provides opportunities for self-reflection and improved understanding. 
Argumentation becomes a means for learners to co-construct knowledge. 


Co-elaboration of new knowledge. Learners participating in argumentation can co- 
construct knowledge that was impossible to construct individually. Argumentation is a 
representation of collaboration because it happens when the group members are willing 
to listen to and work with each other; otherwise, the debate becomes a “war” or 
“winning or losing” game. The perspectives of others can help learners to facilitate their 
learning, which might promote a heuristic learning event. 


Increasing articulation. Articulating arguments precisely is the goal of practicing 
argumentation. The ability to produce a persuasive argument is vital in argumentation, 
and learners can improve their ability to frame arguments that touch their specific 
audience with the appropriate respect and tone. 


Andriessen and Baker (2005) found that to overcome the prevalent perception that argument is 
akin to a war, students need to frame responses as cooperative efforts at meaning-making, as 
opposed to personal victories. Developing a learning environment where students feel 
psychologically safe to both give and receive critique promotes collaboration. Instructors can 
promote healthy interactions by repeatedly emphasizing that analysis of ideas should remain 
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focused on the views themselves, and not on the people espousing them. Furthermore, if 
instructors promote a culture which values the pursuit of truth over ego, this collective pursuit of 
truth can help establish task cohesion, which is a shared commitment among the group to 
achieve a goal (in this case, the purpose of pursuing truth). A further goal in collaborative 
learning is to foster outcome interdependence, where personal goal attainment depends on goal 
attainment by other team members. Outcome interdependence can help make argumentation 
less oppositional. For example, activities like JIGSAW, where different students become 
“experts” on various aspects of an issue, allow episodes of argumentation to be group- 
distributed and requires groups to consider each member’s perspective (Andriessen & Baker, 
2005). 


The Analysis of Computer Support 

One goal of CSCL is to provide possibilities for learners to learn through socially co-constructing 
knowledge (Koschmann, 2002). Therefore, CSCL focuses on the design of social technologies 
that meet the following three fundamental requirements: 1) reconfiguration of computational 
media to make new interactions possible, 2) turning communication into substance, and 3) 
exploring the potential of adaptive media. CSCL should be highly flexible and never become a 
digitized replica of the classroom face-to-face interaction. 


Computer-Supported Collaborative Learning (CSCL) in Practice 

Fu and Hwang (2018) examined research on mobile technology-supported collaborative 
learning from 2007-2016. They evaluated CSCL from five perspectives, including the current 
status and trend of a) research distribution, quantity, and methods, b) learning devices, c) 
research participants and measurement issues, d) collaborative learning group composition and 
strategy application, and e) relations between collaborative learning strategies and 
measurement issues. Most notable were their findings on conceptual cooperative learning 
strategies, which are the most widely used in collaborative learning activities. These learning 
strategies help students develop higher order thinking competence. Moreover, Fu and Hwang 
(2018) stressed that it is essential to design activities that engage students in more meaningful 
and authentic collaborative learning contexts to foster self-regulation and life-long learning. One 
suggestion by Fu and Hwang (2018) was implementing innovative technologies, such as 
wearable devices. Fu and Hwang (2018) suggested that researchers should investigate the 
needs of teachers and employers in the use of collaborative mobile learning approaches 
because there is a lack of mobile collaborative research in the fields of professional 
development and corporate training. In general, the educational affordances of mobile 
collaborative learning are perceived as a useful method of supporting ubiquitous learning, 
facilitating context-based learning, developing self-regulated learning, and fostering cross- 
cultural interaction (Fu & Hwang, 2018). 


Al-Samarraie and Saeed (2018), in a review of 29 studies on cloud computing in collaborative 
learning for blended-learning environments, examined cloud computing tools through three 


categories based on the different utilization purposes. The three groups were synchronized tools 
(e.g., Google Apps, Microsoft, and Dropbox), learning management systems (LMSs) (e.g., 
Moodle and Blackboard), and social networking tools (e.g., Facebook and Twitter for 
interpersonal communication). The authors found out that Google Docs and Google Drive (e.g., 
synchronized tools) serve as the “motivator” to stimulate group members to autonomously 


contribute to the collaborative learning process by allowing them to synchronously and 
asynchronously edit, comment, upload/download files, and share thoughts and experiences. 
LMSs, such as Blackboard, enable students to co-construct knowledge by providing them a 
platform that contains the necessary tool for interaction, instruction, and learning (e.g., 
discussion forum, lecture videos, reading materials, assignments, and quizzes). Other results 
demonstrated that students using Blackboard might find it challenging to construct meanings 
that go beyond the learning context. Additionally, the authors found that the literature supports 
the use of social media tools in increasing students’ satisfaction by significantly facilitating peer 
interactions (e.g., hit the like button and comment on posts). Twitter has become the most used 
form of communication due to its design, such as sharing opinions in the way of tweets that are 
public and visible to everyone, to encourage discussion and social interaction. Despite these 
possible benefits, Al-Samarraie and Saeed (2018) also highlighted some challenges from using 
cloud computing tools, such as the low-technology competence for faculty members, difficulties 
in measuring individual performance versus group performance, and higher technical 
requirements on maintaining and developing cloud computing tools (Al-Samarraie & Saeed, 
2018). 


van Leeuwen and Janssen (2019) reviewed 66 studies about the relationship between teacher 
guidance strategies and the processes and outcomes of collaboration among students in primary 
and secondary education. Their findings showed that, despite the positive relationship between 
teacher guidance and student collaboration, understanding the role of teachers in collaborative 
learning poses some significant challenges. In particular, the importance of the presence or 
absence of the teacher shows mixed results, especially in the meta-level of student activities 
(e.g., focusing on interacting with students about the strategies for solving the task or 
collaboration processes versus answering task-related questions or providing feedback). Van 
Leeuwen and Janssen (2019) suggested this occurs because it is hard for teachers to provide 
prompt feedback at the right moment without taking over students’ collaborative processes and 
causing the feeling of deprivation of autonomy. As a remedy, the authors pointed to particular 
means of teacher guidance that can positively affect student collaboration including helping 
teachers develop competence in providing feedback to students on how to use learning 
strategies and how to make learning plans, and transferring control to students over their own 
learning processes by promoting voluntarily help-seeking (van Leeuwen & Janssen, 2019). 


Liao, Wang, Ran, and Yang (2014) presented a new model of collaborative online learning using 
cloud computing tools to support individual learners’ needs throughout the online learning 
process. After examining the topological structures of online learning, Liao et al. (2014) 
described their nature and their limitations. Three typical interaction patterns emerged, as 
follows: 1) the star-shaped topological relationship (See Figure 1). In this pattern of interaction, 
several students independently connect to a single teacher who is in the center of the 
communications for the class. This traditional e-learning structure prevents students from getting 
timely feedback and increases instructors’ workload, 2) the hierarchy team-based topological 
relationship (See Figure 2). In this typical pattern of interaction, the instructor occupies the top 
tier, and team leaders work under the instructor. The lowest level of the hierarchy contains 
students who are not in a leadership position (Liao et al., 2014). This structure has limitations, 
such as team size, and might cause “free riders” or students that do not adequately participate, 
and 3) the net-based topological relationship (See Figure 3). This interaction pattern shows 


connections between the instructor and the students as free and autonomous. The primary 
constraint of this structure is that individual learners cannot receive enough learning support due 
to the various learning needs that arise in large class sizes. Liao et al. (2014) proposed an e- 


learning solution that integrates cloud computing and e-learning to deliver student learning 


support services. For instance, in a collaborative e-learning environment, the resources are not 
only limited in storage capacity, hardware, servers, and networking components, but also 
human resources such as peers, teachers, and teaching assistants. Liao et al. (2014) further 
stipulated that marketplace rules should be applied to the learning environment to stimulate 
student participation and reasonably dispatch virtual resources among collaborators, such as 
using virtual money (e.g., credit scores) to pay for additional learning support services. As 
shown in Figure 4, their proposed collaborative learning model resembles a cloud-shaped 
topological relationship. Prototype testing of this model of cooperative learning uncovered that 
peer tutoring was acceptable among students, the effects of recommendations generated by the 
algorism of dispatch were positive, and the virtual money and marketplace rules were perceived 
to be helpful by over half of the students (55.7%). 


Figure 1. The Star-Shaped Topological Relationship 


Figure 2. The Hierarchy Team-Based Topological Relationship 


Figure 3. The Net-Based Topological Relationship 


Figure 4. The Cloud-Shaped Topological Relationship 


Furberg (2016), in a secondary education science lab, confirmed patterns consistent with 
previous research on student learning in computer-supported labs. Generally, students faced 
obstacles when trying to link learning obtained through digital lab tools to practical lab work. 
Additionally, teacher support in such settings is critical for improving students’ understanding. 


Team training 
Electronic Team Training: Defining, Strategies, and Effectiveness 


As the world grows in complexity, so does the need for a more efficient way of training those 
that are to operate in this complex world. Similarly, as the tasks increase in complexity, the 
demand for teams of trained individuals is higher than ever before (Robideau & Vogel, 2014). To 
meet the demand for these qualified teams, electronic team training or “e-team training” has 


been on the forefront of many industries over the past decade (Punnarumol, 2015). E-team 
training takes many forms, such as online communication (Han, Chae, Macko, Park, & Beyerlein, 
2017), simulation-based training, in which teams are able to train in person with module 
technology (Leithead et al., 2019), or virtual world training, in which the team trains together 
(Heinrichs, Youngblood, Harter, & Dev, 2008). Implementing e-team training has two benefits. 
The first is the benefits that arise from being on the team in terms of improved performance over 
traditional training methods, such as increased response time, better communication skills, and 
improved confidence and satisfaction reported by team members (Umoren et al., 2017). The 
second benefit is more immediately practical, e-team training can reduction reduce training 
costs for specialized applications and allows training in situations that are difficult to recreate 
(Robideau & Vogel, 2014). To discuss e-team training, team training must first be defined. 


Individual training to teach a person a skill or type of behavior is commonplace; however, the 
growing complexity of tasks today requires individuals to work together in teams, and, 
therefore, be trained as a team. Team training can be defined in a multitude of ways. Taskwork is 
the job itself that, to be done proficiently, requires a team. Teamwork is the ability for people to 
use their individual procedural knowledge as a functioning part of a team, team building 
happens on the worksite and focuses on analyzing a team’s procedures to improve productivity. 
Finally, cooperative learning occurs when a team is taught the initial knowledge together rather 
than being brought together afterwards (Jones et al., 2018). As team training is a complex 
concept with many different components, different team training strategies have emerged to 
combat the challenge. 


Many strategies have emerged to deliver the best possible training to teams. Over the course of 
many studies and analyses, three specific strategies have successfully endured empirical 
scrutiny in the past (Schmutz, Kolbe, & Eppich, 2018). These strategies are cross-training, team 
coordination and adoption training, and guided team-self correction training. 


Cross-training requires team members to rotate positions during training, in order to develop a 
better understanding for the tasks of their fellow team members, experience how each 
individual plays their role in the team, and build a better overall framework for understanding 
the team’s task at hand (Umoren et al., 2017). Understanding each team member’s role can 
improve team efficiency. 


Team coordination and adoption training focuses on training team members to improve their 
coordination strategy by reducing the amount of communication necessary for a given task 
(Haspel et al., 2019). In time sensitive or life and death tasks, losing time to discuss the task at 
hand can ultimately lead to the failure of the team. Thus, training to operate without these idle 
periods of discussion gives the team the most time and highest chance of success on a task. 


Guided team self-correction training refers to a self-reflective event in which team members 
diagnose their team’s problems post-task, and discuss ways to improve (Chamberland et al., 
2018). Regardless of the type of team training strategy that is required, it can be done 
electronically with greater ease than by using traditional training methods. E-team training 
allows these methods to be used in a much safer environment than traditional training would 


allow, without the risk of damaging equipment or suffering loss of life in training (Van De Ven et 
al., 2017). 


In addition to borrowing training strategies from traditional team training, e-team training can 
borrow measures of training effectiveness as well. The way team training has been measured, 
depends largely on the task at hand or what the objective is for the training event. For example, 
in an emergency medical scenario the team would be evaluated on not only if the patient 
survived, but also on their use of time (De Brin, O’Donovan, & McAuliffe, 2019). How much time 
was required for the team to start surgery and how much down time or idle time the team 
exhibited are both key factors in the success rate of an emergency scenario, as well as 
something that can easily be measured in an e-team training environment (De Brin et al., 2019). 
Team training can also be measured in more subjectively reported measures, such as team 
member confidence or satisfaction levels. Previous studies demonstrated that these measures 
correlate with improved team results, therefore, it is imperative that they not be overlooked 
(McLaughlin et al, 2018). 


Improvements in Methods of Traditional Team Training 


E-team training has improved traditional team training methods (Punnarumol, 2015). The 
improvements themselves are largely dependent on what technology is implemented and what 
goals the tasks will help the team achieve. The methods that e-team training utilizes can be split 
into two main categories for this review. These two categories are online communication, in 
which technology is only used for discussion (MacDonald, Stodel, & Casimiro, 2006), and 
simulation training, in which the team members practice their knowledge to learn an applied 
skill (Merién, Van de Ven, Mol, Houterman, & Oei, 2010). 


Online Team Training 


Online communication is a fundamental method of e-team training that is ingrained in most 
industries, so it is overlooked as an improvement on the traditional means of team 
communication. Being able to effectively communicate goals for the team, formulate plans of 
action, and learn from or about new team members can be crucial for effective teamwork and 
team management (Han et al., 2017). 


Implementing basic means of communication by software such as “Skype” can increase 
communication skills by allowing easy and simple access to other team members (Han et al., 
2017). Additionally, the use of online communication in e-team training allows for team 
formation, regardless of physical distance. The need to form a specialized team of experts to 
discuss a problem can pose a logistical nightmare without the use of online communication 
(Plotnick, Hiltz, & Privman, 2016). 


Allowing team members to communicate before and after meetings facilitates other important 
strategies of team training such as team coordination and adoption training and as well as 
guided team self-corrective training. Online communication not only allows for discussions of 
coordination and requirements prior to beginning the task, which helps minimize time during 
the task, but also allows for discussion after the task to go over what needs to be improved 


(Chamberland et al., 2018) However the forefront of e-team training, as well as the strategy of 
cross-training, lies in the second method of simulation-based training. 


Simulation-hased Training 


Simulation-based e-team training promotes improvements in team training that would not be 
practical or even possible with traditional training methods. While simulations make practicing 
some tasks, such as surgery, more practical, they can also make it possible to rehearse tasks not 
normally able to be practiced. Such is the case with crisis response teams that respond to natural 
disasters (Fung et al., 2015). So, from a practical standpoint, simulation e-team training allows 
teams to practice their taskwork, with no personal risk at all to any involved, in situations that 
would normally carry the risk of death (Leithead et al., 2019). In traditional methods, the risk can 
be to the members of the team themselves, such as in military training (Fan & Wen, 2019) or 
aviation (Littlepage et al., 2016). Additionally, the risk could be to the individuals the team is 
using for practice, such as a medical team performing surgery (Naumann, Bowley, Midwinter, 
Walker, & Pallister, 2016). Through the use simulation-based e-team training, the risk is 
eliminated completely for both the team members and those that would normally serve as 
practice participants for the team. 


Given the range of simulations needed in the examples above, different types of equipment are 
required for different e-team training tasks. In situations where the team is physically together, 
the team can utilize e-team training using models or physical items to train with. This can include 
simulation booths featuring a cockpit and ground control set up (Littlepage et al., 2016), ora 
dummy mannequin with synthetic interior organs and sensors for surgery applications 
(Naumann et al., 2016). However, the simulation can also take place completely in a virtual world 
(Heinrichs et al., 2008). These virtual worlds have many benefits and have been shown to 
increase the effectiveness of the e-team training (O’Connor & Menaker, 2008). They grant the 
ability for teams to train together, rather than through separate training modules, regardless of 
the issue of distance or language barriers (Umoren et al., 2017). 


Musharraf and colleagues have found that virtual worlds allow for the use of “synthetic 
teammates” or Al-controlled individuals to populate the world. These synthetic teammates allow 
individuals to participate in team training, even if the members of that team are not available in 
any capacity. Furthermore, as the members of a specific team can change over time, the 
synthetic teammates allow for a neutral method of team training without individual quirks of team 
members influencing that trainees experience (Musharraf, Khan, & Veitch, 2019). 


Improvements Documented in Electronic Team Training 


The implementation of e-team training yields a multitude of benefits for the teams that use them 
as well as for the organizations implementing e-team training (Onan et al., 2017). These benefits 
can be broken down into two categories, observable and self-reported. Observable benefits 
include improved team effectiveness (Heng, Everlyn, & Lateef, 2015), quicker action and 
response times (Murphy et al., 2018), and a reduction in training costs (Allen et al., 2016). Self- 
reported benefits include improvement in communication skills (James, Page, & Sprague, 2016), 


improvement in team leadership (Rosenman, Vrablik, Brolliar, Chipman, & Fernandez, 2019), 
and increased team confidence and satisfaction (Plotnick et al., 2016). 


Team effectiveness is measurable in a variety of ways depending on the task given. With so 
much variance in the way team performance is measured, research in the field centers on the 
need to create assessment tools for team performance (Eppich et al, 2015). For example, 
medical teams in an emergency might measure their team’s effectiveness on whether the patient 
survives. Teams in an industry environment can measure performance in terms of how much 
time was lost in each day by comparing allotted task completion time to actual task completion 
time (Littlepage et al., 2016). Even teams that are working with ongoing projects that have no 
definitive purpose besides system maintenance are assessable by evaluating the number of 
errors made, with a goal of lowering error rates (Jones et al., 2018). All the interventions listed 
above have resulted in improvements in team performance goals using e-team training. 
Alternatively, improving response times of a team in time critical situations is a much simpler 
assessment than appraising team effectiveness. Studies involving the training of teams for 
emergency situations demonstrate that e-team training is beneficial for reducing the amount of 
time the teams require to begin a task, showing faster response times in these time critical 
operations (Murphy et al., 2018). 


Finally, for the observable measures, the financial costs in e-training are much lower than 
training teams using the actual equipment (Littlepage et al., 2016). This reduction in cost occurs 
because e-team training avoids damage to high cost equipment during training, such as an 
airplane in an aviation-based team. Also, cost savings are realized because e-team training 
lowers procurement costs for items such as cadavers in medical simulations (Allen et al., 2016). 
The use of e-team training also lowers logistical wait times involved in procuring or repairing 
expensive items, which can further decrease costs. 


Affective Benefits of Team Training 


Self-reported measures of how members of a team view their ability to communicate, their 
perceptions of team leadership, and their confidence and satisfaction in their team, while less 
immediately quantifiable, are shown to have positive effects on team and member performance 
(Zemliansky, 2012). Self-reported measures of a team’s ability to communicate effectively, or the 
ability to work together with minimal communication result in decreased error rates by the team, 
as well as decreased task completion times (James et al., 2016). Effective team leadership has 
also been linked to improved teamwork and increased team performance. Teams that have 
undergone e-team training specifically geared towards the goal of promoting leadership, have 
found that the impact can last up to 24 months after the completion of the training (Rosenman et 
al., 2019). Lastly team member satisfaction, while seemingly not as imperative as the other 
measures, has been shown to have an impact on the overall quality of the team in long term 
evaluation (Han et al., 2017). Individuals and teams that have had access to e-team training 
techniques have reported an increase in satisfaction with their training and an improvement in 
performance (Plotnick et al., 2016). 


High Risk Areas with Successful Electronic Team Training Implementation 


While the goal of all teams is to succeed, teams that operate in high risk and high stress 
environments demand excellent and authentic team-based training. This is necessary as these 
fields carry with them the potential for the loss of life of a team member or victim (Fan & Wen, 
2019). While this report has mentioned many fields, this section will outline specifically five 
areas of high risk and high stress work that have already seen the successful implementation of 
e-team training. 


The military, being a diverse organization, implements e-team training techniques in a variety of 
ways across departments. Fan & Wen (2019) have conducted research with soldiers that receive 
training using virtual reality simulators that utilize Body Area Networks or BAN that capture their 
movements in real time. These BAN simulators allow multiple soldiers to be present in the 
simulation and for the individuals to see their fellow team members’ movements in real time, 
allowing team tactical training in squad-based movements and team-based tactics. This training 
has been shown to have an improvement over individual simulator training (Fan & Wen, 2019). 


The medical staff of the military have also received e-team training using high-fidelity simulation 
models of trauma victims (Naumann et al, 2016). The need for simulation practice arose as 
trauma teams had increasing numbers of patients that received injuries from improvised 
explosive devices. The simulations have proven to not only be effective for improving teamwork, 
communication, individual decision-making ability, and overall situational awareness, but the 
simulations have been able to provide a much more realistic experience than that of a cadaver 
that cannot easily replicate the issue of bleeding and blood loss (Naumann et al., 2016). The 
benefits of e-team training exist even in the elite multidisciplinary teams such as the U.S. Army 
Forward Surgical Teams (Allen et al, 2016). 


The same benefits that the military’s medical teams have seen from e-team training have also 
been seen in civilian emergency medical teams (Theilen, Fraser, Jones, Leonard, & Simpson, 
2017). Simulation e-team training has also been utilized for the aviation portion of the military. 
The ability to train ground control and pilots together in an e-team training simulation allows for 
the teams to not only practice regular means of coordination and skills, but also allows for the 
simulated practice of emergency flight scenarios without the risk of the pilot or the aircraft 
(Littlepage et al. 2016). Like how the civilian medical teams have shown the same benefits of 
receiving e-team training as the military medical teams, so have the civilian aviation 
departments benefited from the same team-based simulation tools. 


Specialized civilian teams have also found e-team training practices to improve their training. 
One such specialized team is Crisis Intervention Teams or CIT, which is a police and mental 
health collaboration team specializing in involvement of those with mental illness (Crisanti, 
Earheart, Rosenbaum, Tinney, & Duhigg, 2019). As 10-25% of those living with a mental health 
problem have a history of police arrest, reports have shown that the police involvement with 
managing mental health-related cases can take up more time and resources than that of traffic 
accidents, burglaries, or felony assaults (Crisanti et al. 2019). Through the use of e-team training, 
police officers have been able to empower themselves with the knowledge of how to adapt their 
techniques when encountering individuals of this specific nature, and are able to do so without 
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the unethical or dangerous implications of practicing on those people with mental illnesses 
(Stanojevic & Stanojevic, 2016) The e-team training software has been coordinated through 
network hubs that include psychiatrists, crisis intervention unit detectives, and crisis specialists, 
which have started connecting to officers across the country (Crisanti et al, 2019). Using these 
networks of information, and virtual simulation training, officers will be more adept at handling a 
very vulnerable population in the best trained way possible (Stanojevi¢ & Stanojevic, 2016). 


Emergency teams that operate in the event of a natural or man-made disaster must also have 
appropriate training in order prepare themselves (Crichton, Moffat, & Crichton, 2017). This 
training applies to all levels of the team, from the ability to work quickly and effectively in the 
field to being able to make complex logistic decisions in regard to the management of resources 
during the emergency (Fung et al. 2015). Training with how to identify those that are need of 
help, and how to identify those that can be helped in an emergency takes specialized training 
that can make the difference in terms of maximizing the number of surviving patients (Heinrichs 
et al., 2008). Having the ability to simulate many different types of disasters in a cost-effective 
manner allows for more accessible training to emergency department teams, over that of the 
traditional training scenarios which are costly and time consuming to attempt to recreate 
(Heinrich et al. 2008). Additionally, the e-team training technology gives the benefit of 
immersive virtual worlds to better put the team into the scenario of the emergency (Liaw, 
Carpio, Lau, Tan, Lim, & Goh, 2018). 


Additional Areas with Successful Electronic Team Training Implementation 


Teams are not always operating in time sensitive, high risk, and high stress environments, but 
any type of team can benefit from e-team training. Such training has seen to be greatly beneficial 
to those in business, technical work, and physical labor jobs. Business entities such as the 
infamous Six Sigma organization have implementing the use of e-team training tools to improve 
workshops on roles and responsibilities, standard workflow, and leadership training for some 
time now (Albertson, 2019). While not utilizing as much simulation technology as the other fields 
of work, the use of e-team training in even its most basic forms of online communication and 
team building have been shown to improve group performance on creativity, communication, 
dynamic group-feedback, and overall success (Han et al., 2017). The improvement team 
dynamics that would normally be limited by issues of physical distance have benefited from e- 
team training (Pennington et al., 2018). 


Industry professionals have also benefited from implementing e-team training in their 
organizations. Simulation-based training has been shown to improve training safety and 
reliability for teams in possibly hazardous environments (Musharraf et al., 2019). One such 
industry that has adapted to the e-training method is the drilling industry, in both 
offshore/onshore as well as for the use of technical and non-technical knowledge positions 
(Crichton et al. 2017). The use of e-team training in high fidelity drilling simulators has led to 
increased opportunities to practice, test out, and receive feedback on work skills during the 
training process, which has led to increased team performance (Crichton et al., 2017). While not 
the focus, the e-team training is also utilized for providing emergency situation training to the 
drillers, in the unlikely event that there is an accident at the workplace (Musharraf et al., 2019). 


Blended Learning and Team Training 

Holmes and colleagues note that team building has emerged as a factor in higher education in 
blended doctoral programs. Hybrid doctoral programs are different in character than traditional 
programs and team building has been suggested as a mechanism to advance student 
persistence. Holmes and colleagues (2014) found that team building, and collaborative exercises 
highlighted the strengths of each team member’s contribution to the learning cohort. Each team 
member in the cohort brought their own skills to the tasks and this equated to a division of labor. 
The team in their experimental study demonstrated a sense of definable membership, 
awareness of one’s membership on the team, member interaction, a shared sense of purpose, 
and the awareness to act individually and as a unit. Working together helped students overcome 
isolation, encouraged cooperation, and promoted cohesion on group projects. The process of 
team building occurred in the hybrid doctoral program through communication, duties 
coordination, and member feedback (Holmes, Trimble, & Morrison-Danner, 2014). 


Sonnesson and colleagues have found that blended learning has also been successfully used in 
advanced trauma training in both civilian and military contexts). The biggest difficulty in 
learning to manage trauma was the lack of authentic practice on difficult cases. Practicing real- 
world reasoning and decision making on virtual patients through blended learning supported 
the education and training of the professionals and had benefits for improving pre-deployment 
training with reduced strain on home civilian hospitals. 


Team Training in MO0Cs 


The use of MOOCs for team training can be initially difficult, as one of the draws of MOOCs is the 
ability to get online and learn without any formal times or locations, which makes team training 
or team-based learning a challenge to implement, as these cooperative tasks may not even be 
desired by the learners using MOOCs (Verstegen, Dailey-Hebert, Fonteijn, Clarebout, & Spruijt, 
2018). This has been identified as an obstacle, rather than a user preference, as collaborative 
learning has been shown to increase the effectiveness of MOOCs when able to be utilized (Sanz- 
Martinez, Martinez-Monés, Dimitriadis, & Bote-Lorenzo, 2019). As such, the formation of teams in 
MOOCs must be handled with care, as the status of MOOCs lacks these collaborative team 
training exercises (Spoelstra, van Rosmalen, Houtmans, & Sloep, 2015). Some universities have 
already found some success with team-based learning in MOOCs; however, some have noted 
that there is some increase in the technical aspects of set-up and cost (Mayagoitia & Varela, 
2019). 


Retrieval practice/testing effect 


The Testing Effect and Retrieval Practice 


What is Retrieval? 

The human memory works in a somewhat circular fashion such that the key to maintaining 
knowledge in the memory for later use is maintaining access to the knowledge, and the key to 
maintaining access to the knowledge is to use it frequently (Bjork, 1988). The very act of 
recalling the information, called retrieval, is critical to subsequent successful access to the 
knowledge (Bjork, 1988; Brown, Roediger, & McDaniel, 2014). Retrieval practice is the periodic 
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recalling of facts, concepts, or life events from memory (Brown et al., 2014). The benefit of 
frequent retrieval practice is not only a matter of improving the representation of the facts or 
events in the memory, but the act of retrieval also strengthens the process and success of later 
retrieval efforts (Bjork, 1988; Wissman & Rawson, 2017). This phenomenon, which is thought to 
be due to the influence that retrieval has on encoding, is called test-potentiated learning 
(Wissman & Rawson, 2017). Kornell, Hays, and Bjork (2009) found that even when retrieval is 
unsuccessful, the effort alone strengthens learning if feedback is given and when the retrieval is 
effortful. Retrieval can take on various forms, such as reflection, questioning, homework, or 
testing (Agarwal, Roediger, McDaniel, & McDermott, 2018). 


Retrieval practice has been successful, not just in the laboratory, but also in the classroom 
(Agarwal, Bain, & Chamberlain, 2012; McDaniel, Agarwal, Huesler, McDermott, & Roediger, 
2011). In educational settings, tests are usually considered as a means of student assessment; 
however, testing has positive effects on subsequent learning too (Roediger & Karpicke, 2006a). 
In studies by Agarwal, Finley, Rose, and Roediger (2017), retrieval practice improved 
performance in all conditions, with or without feedback, and with lag times between study 
opportunities and the initial testing, especially in students with a lower working memory 
capacity. 


What is the Testing Effect? 


The testing effect is a concept that refers to the gains in student learning and retention that 
accompany the testing and retesting of identical material (Adesope, Trevisan, & Sundararajan, 
2017; Butler & Roediger, 2008; Kang, McDermott, & Roediger, 2007; Odegard & Koen, 2007; 
Roediger & Marsh, 2005; Roediger & Karpicke, 2006a). Frequent testing improves performance 
across all levels of education (Roediger & Karpicke, 2006b). 


Testing can have indirect (mediated) or direct (unmediated) effects on learning (Roediger & 
Karpicke, 2006b). Mediated effects are actions that influence the learner and promote learning, 
such as feedback, the act of studying material more consistently over time versus massed 
practice (cramming), and self-testing to determine what concepts have been mastered (Adesope 
et al., 2017; Roediger & Karpicke, 2006b). The unmediated effect is inherent in the act of taking 
the test. Taking tests influences learning and improves long-term retention (Roediger & 
Karpicke, 2006b). 


Studies of the testing effect are often conducted around a testing format, such as free recall, cued 
recall, multiple-choice, short answer, or recognition questions (Adesope et al., 2017). Although 
the cognitive processing needed for one testing format (a multiple-choice test, for example) is 
different than that needed for another type of test (cued recall or short answer, for example), the 


testing effect remains strong for all testing scenarios (Adesope et al., 2017). A significant 


moderator of the strength of the testing effect is when transfer appropriate processing (TAP) is in 
place in the test/retest scenario (Adesope et al., 2017; Morris, Bransford, & Franks, 1977). 
Transfer appropriate processing is the condition in which the pretest and posttest are identical 
such that the mental processes needed to negotiate the pretest are the same as those used to 
complete the posttest (Adesope et al., 2017). Transfer appropriate processing necessitates that 
the information is presented in the context in which it will be tested. For example, a multiple- 
choice pretest is followed by a multiple-choice posttest (Morris et al., 1977). Hays, Kornell, and 
Bjork (2010) and Thomas, Weywadt, Anderson, Martinez-Papponi, and McDaniel (2018) disputed 
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that the test format must be identical and believe that the TAP theory may be too simplistic. In 
their experiments, Hays et al. (2010) compared the effect of an initial short answer test followed 
by a multiple-choice posttest and found that memory durability is stronger when the initial test 
requires greater effort. This effect was substantiated by Pyc and Rawson (2008) with the addition 
of a caveat that when retrieval is no longer difficult, the benefit profile shows a law of diminishing 
returns on test performance. 


Below is a table based on the conclusions of a meta-analysis of the testing effect, performed by 
Adesope et al. (2017), that shows the strength of the testing effect for different question types. 


Effect Sizes of Different Question Formats on the Testing Effect 


Weighted Mean Effect Sizes Format of Practice Test Format of Final Test 
expressed as Hedges g 


a a 
a a 


From the table above, it is noteworthy that multiple-choice questions exhibit the largest effect 
sizes when they are used as a question type in pretesting. This may be due to the lower 
processing demands associated with that question type; higher cognitive demands are needed 
to answer free-recall, short answer, or cued-recall questions (Adesope et al., 2017; Morris et al., 
1977). Despite the large effect size found with utilizing multiple-choice questions, caution must 
be used with the data. For example, multiple-choice questions can have positive and negative 
effects on long-term retention (Roediger & Marsh, 2005). Even though the testing effect is nearly 
universally evident, using multiple-choice questions in pre-testing can result in students reading 
the misinformation (lures) and, in no-feedback conditions, the lures can corrupt student 
performance on post-tests. True/false tests also expose students to misinformation (Roediger& 
Marsh, 2005). Toppino and Brochin (1989) found that the act of exposing students to false 
answers increased the belief that the wrong choices were true on subsequent testing. Quizzing 
with factual questions improved application-type examination transfer and quizzing with 
application questions improved performance on factual-type exams (Thomas et al., 2018). 
Roediger & Marsh (2005) found that increasing multiple-choice alternatives from two to four 
decreased the proportion of correct answers chosen; however, adding six alternatives did not 
significantly affect the students’ scores. Furthermore, adding a “none of the above” option as the 
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correct choice negated the testing effect (Odegard & Koen, 2007). The testing effect remained if 
“none of the above” was the incorrect answer (Odegard & Koen, 2007). 


Feedback can reverse the negative effects of the misinformation in the multiple-choice lures 
(Butler & Roediger, 2008), although feedback is not universally beneficial in all scenarios. For 
example, Hays et al. (2010) found that when there were time constraints for learning, students 
skipped unnecessary feedback in favor of more retrieval opportunities with no detrimental effect 
on their learning. Thomas et al. (2018) found quizzing plus feedback prior to examination 
showed student improvement versus no-quizzing conditions. 


The Learning Effects of Testing versus Other Conditions 

Adesope et al. (2017) have shown that taking practice tests shows effect sizes that are stronger 
than those found from no study or from utilizing filler activities that were off topic relative to the 
final test (g = 0.93). Adesope et al. (2017) also found that taking practice tests had a greater 
effect size than other retrieval activities such as rereading or re-studying (g = 0.51). Cautioning 
that the no study and unrelated filler activities comparison is likely due to those activities not 
offering related retrieval and urges that, for educators, the comparison between testing and 
other relevant retrieval activities is more accurate (g = 0.51). 


Features of practice tests 
The testing effect can be modified by adjusting the number and frequency of the testing 


episodes (Carpenter & DeLosh, 2005). Many researchers have found evidence of the spacing 
effect, which is defined as testing information to be learned in a distributed fashion rather than in 
a massed fashion. Additionally, Carpenter and DeLosh (2005) found that expanded retrieval did 


not improve retention at testing; however, Cull, Shaughnessy, Zechmeister (1996) contradicted 
these findings in favor of an expanding form of retrieval such as increasing intervals between 
retrieval efforts. In addition to the frequency evidence, there is evidence that taking more 
practice tests can improve encoding during re-study (Wissman & Rawson, 2017). 


In online courses, testing has been shown to be beneficial to reduce mind-wandering, increase 
notetaking, in addition to facilitating learning (Szpuner, Khan, & Schacter, 2013). Walck-Shannon, 
Cahill, McDaniel, and Frey (2019) found that opening previously scored online quizzes for 
students to study prior to post-testing or final examinations improved performance on both tests. 
Interestingly, these authors also found that students with lower STEM scores in previous courses 
benefited more from the voluntary re-quizzing than did their higher-performing counterparts on 
the final examination, but not post-testing (Walck-Shannon et al., 2019). 


Pan and Rickard (2018) documented that the transfer value of the testing effect in their meta- 
analyses of test-enhanced learning. Pan and Rickard (2018) found that transfer strength is 
dependent upon three constructs: the congruency of the pretest and posttest answers, the use of 
elaborated feedback during training, and the level of performance on the initial test. When there 
is a strong congruency between pre- and posttest answers, learning transfer was observed. 
Likewise, when elaborated feedback is used in training, transfer was more likely. Finally, when 
retrieval was successful on the pretest, transfer was more likely (Pan & Rickard, 2018). Adesope 
et al. (2017) found that that there was no significant difference in the testing effect between 
retention testing (g = 0.63) and transfer testing (g = 0.53). 


Boundary conditions of the testing effect 

Boundary conditions of the testing effect are found when the testing format, success of initial 
retrieval effort, retention interval, and spacing of tests are manipulated (van Gog & Sweller, 
2015). Other potential boundary conditions are the type of retrieval and the complexity of the 
material (van Gog et al., 2015). 


In a study by Sundqvist, Montyla, and Jonsson (2017), a small effect size was demonstrated for 
overt retrieval, which they defined as retrieval coupled with articulation of the knowledge, 
versus covert retrieval (retrieval without articulation). The benefit of covert retrieval was also 
demonstrated by Carpenter, Pashler, and Vul (2006). Wissman and Rawson (2017) provided 
evidence against the covert retrieval hypothesis. 


Complex learning tasks are those with high element interactivity, and therefore, are manifested 
in higher intrinsic and extraneous cognitive load being placed on the learner (Sweller, 2010). 
Both low- and high-complexity tasks are necessary to learning such that different loads are 
placed on learners in educational settings (van Gog et al., 2015). Since must educational 
psychology research is carried out using cued recall, van Gog and Sweller (2015) asserted that it 
does not adequately mimic the complexity of educational materials used in schools. Van Gog et 
al. (2015) argued that the testing effect must be evaluated in both types of element interactivity. 
Van Gog et al. (2015) found that the testing effect was not evident on final test performance when 
problem-solving skills were acquired through worked examples. 


Competency-based Learning 

The Department of Defense is responsible for training and educating personnel to a minimum 
level of proficiency (Smith, Hernandez, & Gordon, 2019). Traditionally, there has been a 
separation between these two entities, with education focusing more on incrementally 
understanding concepts using reflection, while training has been associated with readiness and 
skill that can be demonstrated and for which there is immediate feedback. Credentialing is 
generally used to evaluate personnel and permanently qualify them for service; however, in 
some sectors, such as healthcare, aviation, or nuclear power, constant re-evaluation is required. 
Although studies demonstrate that 70-90% of learning happens on-the-job, there is no credible 
way to catalog those learning activities. Often, there is a lack of well-defined performance 
indicators based on credentials that supervisors can apply to track, and, subsequently, there is a 
lack of predictive capability of how personnel will perform over the course of their careers. 
Smith et al. (2019) noted that lack of readiness in performance becomes evident only after a 
major problem or crisis arises. 


Competency-based learning (CBL) is similar to competency-based education, but accounts for 
the unique training that occurs in military contexts that encompass the service members 
knowledge, attitude, skills, traits, abilities, and other aptitudes (Smith et al., 2019). Ina sense, the 
military training is seen as a complementary endeavor to the service members traditional formal 
education (Smith et al., 2019). In traditional learning, time is the constant and performance is 
variable; however, in competency-based learning, time becomes the variable and performance 
becomes the constant, and the source of learning is less important than mastery of the material 
demonstrated in performance (Stafford, 2019). 


Stafford (2019) found that competencies are not arbitrary but are groups of capabilities that are 
transferable across a range of performance requirements. Further, competencies are usually 
composed of institutional or core competencies that reflect the values of the institution, and 
occupational or specialty competencies that reflect the intricacies of a specific vocation or job. 
Competency-based learning is less common in education than in training because of the 
difficulty of extrapolating competencies from purely cognitive development. Nevertheless, 
cognitive competencies like analytical thinking, conceptual thinking, critical thinking, 
commitment to learning, and diagnostic skills are foundational to educational outcomes (Stafford, 
2019). 


The foundation of competency-based education is revealed in its definition. In 2011, more than 
100 stakeholders in competency-based education formulated a working definition which 
includes the following precepts: 


1) Students advance through stages of learning as mastery is demonstrated. 

2) Competencies are explicit, measurable, and transferable learning objectives that enable 
students. 

3) Assessment is both a meaningful and positive learning experience. 

4) Students receive feedback that is timely and individualized to maximally support their 
increasing development. 

5) Learning outcomes include application of knowledge and creation of knowledge, in 
conjunction with the development of critical skills and dispositions (Sturgis & Casey, 
2018). 


Sturgis and Casey (2018) defined distinctions of CBL compared to traditional educational center 
around differences in outcomes, mindset, culture, support, pedagogy, assessment, reliability, 
learning, infrastructure, grading, and advancement. For example, in the traditional learning 
environments, the full range of a student’s skills are neglected. Softer skills like social and 
emotional skills are often ignored, and application of skills is often undervalued. However, social 
and emotional intelligence are foundational to learning. In contrast, CBL encourages students to 
use their prior knowledge and abilities as a platform to develop knowledge, skills, and 
competencies. 


Another way that Sturgis and Casey (2018) demonstrate how CBL differs from traditional 
educational formats concerns the mindset that a student’s aptitude is fixed and unchangeable. In 
contrast, CBL promotes a growth mindset that meets students in their current knowledge base 
and seeks to advance a student’s learning to a more mastery level by addressing knowledge or 


skill gaps. In many such ways, CBL and personalized education are complementary. 
Competency elevates instruction from a ‘“‘one-size-fits-all” format to an individual learning 
pathway that is student controlled. Sturgis and Casey (2018) argued that attempts to personalize 
education without CBL will promote a wider disparity in educational equality because CBL 
ensures that educational success is tied to concrete standards that can be monitored and 
supported as the student progresses to mastery, thus supporting equality in demonstrating 
knowledge. They also proposed developing a CBL environment following sixteen design 
principles that are divided into three categories, namely, purpose and culture principles, 
teaching and learning design principles, and structure design principles. Purpose and cultural 
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principles can be further deconstructed to encompass a driving purpose of the CBL, equitable 
education, nurturing inclusivity, and learning, fostering a growth mindset, and cultivating 
leadership. Teaching and design principles can be broken down into elements such as following 
current learning science principles, activating self-efficacy (student ownership of education), 
development of higher-level thinking skills, and responsiveness of the environment. Structure 
design principles include seeking alignment in learning tasks, transparency, ensuring 
consistency and reliability of the CBL environment, professional development, organizational 
flexibility, continuous organizational improvement, and advancement upon mastery (Sturgis & 
Casey, 2018). 


The theoretical basis for CBL has shown that it is an effective pedagogical practice (Henri, 
Johnson, & Nepal, 2017). CBL has proven effective for learning in interdisciplinary environments 
because it likely leads to more learner autonomy, which is positively related to student 
achievement (Fazey & Fazey, 2001; Henri et al., 2017). Learner autonomy has been positively 
linked to student motivation with research demonstrating that perceived control of their own 
education improves learners’ performance (O’Reilly, 2014). Autonomous students are better 
positioned to reach competency levels in their field and are better able to integrate knowledge 
and information (Henri et al., 2017). Radovan and Makovec (2015) found that there was a high 
correlation between control beliefs, self-efficacy, and goal orientation. Specifically, regarding 
goal orientation and enjoyment of the course, the most impactful factors were teacher support, a 
perception of the usefulness of the course topics, and a perception of autonomy (Radovan & 
Makovec, 2015). Fazey and Fazye (2001) encouraged instructors to facilitate autonomy and to 
deliver high quality instruction to help develop each learner’s potential. 


Successful implementation of CBL can be attained by providing user-friendly, real-time mapping 
tools to help guide curriculum (Wong, 2019). Additionally, teacher support must proactively 
support faculty development. Online earners must be supported through excellent teacher- 
student connectivity, maintaining an appropriate teacher-learner relationship despite distance 
barriers, asking “what if’ questions in clinical settings, allowing students to directly observe 
practitioners, creating feedback opportunities, and encouraging heterogeneity in learner 
groups. 


Henri et al. (2017) suggested that CBL is especially beneficial in STEM education because by 
increasing the student’s autonomy and motivation, in addition to improving positive perceptions 
of learning, student achievement in these subjects can be improved. CBL has been utilized in 
medical, dental, occupational therapy, and other health professions education (Wong, 2019). 
Henri et al. (2017) urged that research studies should be designed that can quantitatively and 
qualitatively examine CBL programs and their impact on student performance. Such research 
should strive for consistency in metrics so that other researchers, designers, and instructors 
could have more confidence in results and anecdotal claims about CBL. 


Blended Learning and CBL 


Cremers, Wals, Wesselink, and Mulder (2016) stated that higher education institutions are 
charged with creating professionals who can solve problems across disciplines, professions, and 


perspectives and hails blended learning as a method for teaching competencies that will 
adequately equip today's workers. Blended learning for developing competency is transforming 
medical education. Maza et al. (2016) were able to transform a competency-based medical 


education course into a blended format while retaining the ability to teach to mastery. They 
found that students promoted flexibility and autonomy in the learning process for students and 
students were able to develop cognitive, procedural, technical, integrative, professional, 
communicative, and reflective competence. 


M00Cs and CBL 


MOOCs have been previously described as a means of competency-based learning, in addition 
to for higher-education and open-ended initiatives (Blackmon, 2018). These competency-based 
practices have ranged from the STEM fields of study to broader topics such as bettering a 
learner’s understanding of climate change (Otto et al., 2019). Competency-based practices in 
MOOCs face the challenge of creating a wider acceptance of their competency accreditations 
that are awarded to their learners, and convincing educational leaders to allow for a transfer of 
their accreditations to count for credit in other institutions (Corlett, 2014). However, the 
empirical research regarding MOOCs and the competency-based learning is still lacking; 
despite several search attempts only minimal information has been found. Further research in 
this area is needed. 
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Appendix D: Survey Results 


Implementation of State-of-the-Art Practices 
Overview 


The state-of-the-art principles were used to create a survey to target perceived importance and 
perceived implementation. This survey specifically aimed to determine expert’s views on the 
importance of the best practices and the extent to which these practices were implemented 
within their organization. It was distributed to a range of organizations spanning the public, 
private, and academic sectors. Our overall results show an interesting pattern of reported 
incorporation within organizations lagging perceived importance. However, this difference was 
only significant within the public sector (which only included military respondents). Our detailed 
results are provided below with graphs. The tables of means and standard deviations are 
provided in Appendix D. The full survey can be found in Appendix E. 


Results 

A 2 (Sector: Military, Civilian) x 2 (Rating: Importance, incorporation) mixed ANOVA was 
conducted for each paired of the question categories (Technology(e.g., intelligent tutoring 
system or video), Technology Features (e.g. personalization, feedback), Instructional 
Methods(e.g., at scale, blended learning, synchronous eLearning), & Supporting Features (e.g., 
memorization, collaboration)) investigate the differences between the perceived importance 
and the perceived incorporation by military and civilian respondents. Results divided by 
category are described below. 


Technology 

The results for the section reporting on the different technologies used in the eLearning 
environment yielded a significant interaction effect, F (1, 14) = 9.67, p = .008; np2 = .41. This 
resulted from the military respondents’ importance ratings being significantly higher than 
perceived incorporation which was not the case for rating in the civilian population. There was a 
main effect of incorporation versus importance, F(1,14)=24.75, p=.000; np2=.64, such that the 
mean score was significantly higher for importance (M = 4.70, SD = .32) than for incorporation (M 
= 3.95, SD = .92). 


Importance vs Incorporation of Technology 


Mean Perception 


Military Civilian Total 


HImportance M#Incorporation 


Figure 1. Mean ratings of the importance and incorporation of technology in e-learning environments. 


Technology Features 

The results for the second section reporting on the different features of technology used for 
enhancing the eLearning environment yielded a significant interaction effect, F (1, 14) =35.69, 
p=.000; np2=.72. This resulted from the military respondents’ importance ratings being 
significantly higher than perceived incorporation which was not the case for rating in the civilian 
population. There was a main effect of incorporation versus importance, F(1,14)=104.50, p=.000; 
Np2=.88, such that the mean score was significantly higher for importance (M = 5.23, SD = .51) 
than for incorporation (M = 3.51, SD = 1.17). 
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Figure 2. Mean ratings of the importance and incorporation of the features of technology. 


Instructional Methods 

The results for the third section reporting on the different instructional methods used for 
successful eLearning did not yield a significant interaction effect, F (1, 14) =3.14, p=.098; np2=.18. 
There was a main effect of incorporation versus importance, F(1,14)=23.72, p=.000; np2=.63, such 
that the mean score was significantly higher for importance (M = 4.58, SD = .60) than for 
incorporation (M = 3.83, SD = .62). 
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Figure 3. Mean ratings of the importance and incorporation of different instructional methods. 


Supporting Features 

The results for the fourth section reporting on the different eLearning principles used to support 
effective eLearning yielded a significant interaction effect, F (1, 13) =21.38, p=.000; np2=.62. This 
resulted from the military respondents’ importance ratings being significantly higher than 
perceived incorporation which was not the case for rating in the civilian population. There was a 
main effect of incorporation versus importance, F(1,13)=57.11, p=.000; np2=.82, such that the 
mean score was significantly higher for importance (M = 5.08, SD = .46) than for incorporation (M 
= 3.93, SD = 1.06). 
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Figure 4. Mean ratings of the importance and incorporation of learning principles to support effective e-learning. 


Description of Respondent's Organization 

The average means for military versus civilian responses in describing their learning 
organization are shown in graph E. The means and standard deviations for all questions can be 
found in Appendix D. 


Respondents were asked to describe their learning organization. Overall, ratings for civilian 
organizations (M = 4.82, SD = 1.32) were higher than for military organizations (M = 3.54, SD = 
1.37) 


Both military and civilian organizations’ mean scores indicate that the respondents agree with 
the following statements that describe their learning organization; understands how people 
learn, values learning, provides opportunities for learning, provides support for learners, fosters 
trust within the organization, and implements in-person (face to face) instruction. 
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Figure 5. Mean ratings of how respondents described their learning organizations, questions one through six. 


The military organizations’ mean scores indicate that the respondents disagree, while the 
civilian organizations’ mean scores indicate that the respondents agree with the following 
statements that describe their learning organization; provides support for instructors, 
communicates well internally about learning, promotes communication between learners and 
instructors, uses course data to evaluate classes, collects user data to understand instructors 
needs, & collects user data to understand student needs. 
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Figure 6. Mean ratings of how respondents described their learning organizations, questions seven through twelve. 


Survey Findings Summary 

Aconsistent theme was found in ratings of the perceived importance and incorporation of 
eLearning features in their organizations. Participants rated the importance of methods and 
principles for eLearning significantly higher than they rated for the successful incorporation of 
those same methods and principles in their organization. Furthermore, the significant interaction 
of three out of the four paired sections suggests that the differences in perceived incorporation 
versus perceived importance is much greater for military organizations than their civilian 
counterparts. This suggests that although the experts within military organizations are aware of 
the best methods and principles for eLearning, the incorporation of these principles lags far 
behind civilian organizations. However, because of the high acceptance of the best 
principles in the public section (military specifically), it may be ready for the transition to 
advanced distributed learning at scale. 


Principles such as learning styles and brain-based learning are still prevalent despite the 
evidence that proves these principles are a fallacy. A strong emphasis on replacing these 
learning fallacies with the new emerging research-based principles will need to be pursued. 
This could best be accomplished by focusing more emphasis on understand basic learning 
principles within the culture of learning organizations which would require top down support. 


Survey Tables of Means 


Tables 1 through 10 present means and standard deviation of individual survey question. 


Table 1. Means and Standard Deviations for participants ratings on: Please read the following 
statements and think about how well they describe your learning organization. Please rate how 
much you agree or disagree with each statement. 


n M | SD 
My organization understands how people learn 16 | 5.13 | 0.89 
My organization values learning 16 | 5.56 | 0.51 
My organization provides opportunities for learning 16 | 5.13 | 0.81 
My organization provides support for learners 16 | 4.88 | 1.20 
My organization provides support for instructors 16 | 4.25 | 1.65 
My organization communicates well internally about learning 16 | 4.19 | 1.42 
My organization promotes communication between learners and instructors 16 | 4.31 | 1.35 
My organization fosters trust within the organization 16 | 4.75 | 1.39 
My organization uses course data to evaluate classes 16 | 4.00 | 1.41 
My organization collects user data to understand instructor needs 16 | 3.56 | 1.59 
My organization collects user data to understand student needs 16 | 3.69 | 1.62 
My organization implements in-person (face to face) instruction 16 | 4.56 | 1.79 


Table 2. Means and Standard Deviations for participants ratings on: Organizations often use 
several technologies for learning. Please rate how often your learning organization incorporates 


the following technologies in the learning environment. 


n M | SD 
telecommunication (e.g., Zoom, Skype, videoconference) 16 | 4.25 | 1.34 
web-based instruction 4.75 | 1.18 
computer-based instruction (e.g., instructional software) 4.44 | 1.21 
simulations, virtual reality, or augmented reality 3.75 | 1.13 


mobile apps 3.63 | 1.31 
social media platforms (e.g., Facebook, Yellowdig) 3.38 | 1.20 
video platforms (e.g., YouTube, Vimeo) 4.13 | 1.36 
interactive systems (e.g., intelligent tutoring systems, dynamic etexts, 
interactive computer-based learning systems) 8.25 | 1.34 
synchronous computer-supported collaborative learning (e.g., a chat room) 3.53 | 1.30 
asynchronous computer-supported collaborative learning (e.g., a discussion 
forum 4.44 | 1.46 


Table 3. Means and Standard Deviations for participants ratings on: In your personal opinion, 
please rate the importance of the following technologies for successful e-learning. 

n M| SD 
telecommunication (e.g., Zoom, Skype, videoconference) 15 | 5.07 | 0.70 
web-based instruction 16 | 5.19 | 0.66 
computer-based instruction (e.g., instructional software) 16 | 4.88 | 0.72 
simulations, virtual reality, or augmented reality 16 | 4.50 | 0.52 
digital educational games 16 | 4.56 | 0.73 
mobile apps 16 | 4.63 | 0.62 
social media platforms (e.g., Facebook, Yellowdig) 16 | 3.81 | 0.98 
video platforms (e.g., YouTube, Vimeo) 16 | 5.25 | 0.58 
interactive systems (e.g., intelligent tutoring systems, dynamic etexts, 
interactive computer-based learning systems) 16 | 4.50 | 0.63 
synchronous computer-supported Collaborative learning (e.g.,a chat room) | 16 | 4.50 | 0.82 
asynchronous computer-supported collaborative learning (e.g., a discussion 
forum) 16 | 4.88 | 0.81 


Table 4. Means and Standard Deviations for participants ratings on: Think about the features of 
the technologies that your organization uses for learning. Across various technologies, please rate 
how often the following features play a role 


adaptivity (e.g., just-in-time feedback and recommendations) 

artificial intelligence (e.g., algorithms to analyze user data) 

tutorials (e.g., prepare learners to use the technology) 

personalized learning (e.g., customization, interactive learning environment, 


flexible scheduling and pacing, and authentic assessment) 
dashboards (e.g., data visualization of student performance) 
video (recordings or streaming) 

usability evaluations 


Table 5. Means and Standard Deviations for participants ratings on: In your personal opinion, 
please rate the importance of the following features for enhancing e-learning. 
n 


adaptivity (e.g., just-in-time feedback and recommendations) 

artificial intelligence (e.g., algorithms to analyze user data) 

tutorials (e.g., prepare learners to use the technology) 

personalized learning (e.g., customization, interactive learning 
environment, flexible scheduling and pacing, and authentic assessment) 
dashboards (e.g., data visualization of student performance) 

video (recordings or streaming) 

usability evaluations 


Table 6. Means and Standard Deviations for participants ratings on: Organizations often use 

several instructional methods for learning. Please rate how often your learning organization 
incorporates the following types of instruction in the learning environment. 

n M | SD 


in-person instruction 16 | 4.69 | 1.30 
asynchronous online learning 16 | 5.00 | 1.03 
synchronous online learning 16 | 3.56 | 1.21 
flipped classrooms 16 | 3.19 | 0.83 
MOOCs (massive open online courses) 16 | 2.50 | 1.32 
mobile learning (technology supported out of classroom learning) 16 | 3.88 | 1.46 
technology-enhanced classrooms 16 | 4.00 | 1.55 


Table 7. Means and Standard Deviations for participants ratings on: In your personal opinion, 
please rate the importance of the following types of instruction for successful e-learning. 

n M | SD 
in-person instruction 16 | 4.31 | 1.35 
asynchronous online learning 16 | 5.38 | 0.62 
synchronous online learning 16 | 4.81 | 1.11 
flipped classrooms 16 | 4.06 | 0.93 
MOOCs (massive open online courses) 16 | 3.63 | 1.50 
mobile learning (technology supported out of classroom learnin 16 | 5.19 | 0.66 
technology-enhanced classrooms 16 | 4.69 | 0.95 


Table 8. Means and Standard Deviations for participants ratings on: To what extent are the 
following principles used to support effective e-learning in your organization? 
n M | SD 
memorization 15 | 4.80 | 0.78 
motivation 15 | 4.53 | 0.83 
critical thinking 15 | 4.33 | 1.11 
problem-solving 15 | 4.60 | 1.18 
feedback 15 | 4.33 | 1.18 
self-evaluation 15 | 3.80 | 1.08 
goal setting 15 | 3.80 | 1.15 
learning styles 15 | 4.27 | 1.71 
brain-based learning 15 | 3.93 | 1.28 
personalized learning 15 | 3.80 | 1.08 
competency-based learning (e.g. demonstrated mastery of skills that make 
up a domain area) 15 | 3.60 | 1.40 
social learning 15 | 3.40 | 1.24 
assessment and testing 15 | 5.33 | 1.29 
team training 15 | 3.47 | 1.41 
collaborative learning 15 | 3.67 | 1.29 
observational learning (e.g., modeling, how to videos) 15 | 4.40 | 1.30 

establishing trust in the content 15 | 4.00 | 1.56 
student support services (e.g., advising, counseling, libraries) 15 | 3.53 | 1.85 


Table 9. Means and Standard Deviations for participants ratings on: In your personal opinion, 
please rate the importance of the following principles for e-learning? 

n 

memorization 15 

motivation 15 

critical thinkin 15 

problem-solving 

feedback 

self-evaluation 

goal setting 

learning styles 

brain-based learning 

personalized learning 

competency-based learning (e.g. demonstrated mastery of skills that make 

up a domain area) 

social learning 

assessment and testing 

team training 

collaborative learning 

observational learning (e.g., modeling, how to videos) 

establishing trust in the content 

student support services (e.g., advising, counseling, libraries) 

Table 10. 

Means and Standard Deviations for participants ratings on: Please read the following statements 

and think about how well they describe your learning organization. Please rate how much you 

agree or disagree with each statement. 


military 


civilian 


M 


SD 


M 


My organization understands how people learn 


4.25 


0.50 


5.42 


My organization values learning 


5.00 


0.00 


5.75 


My organization provides opportunities for learning 


4.25 


0.50 


5.42 


My organization provides support for learners 


4.00 


1.41 


5.17 


My organization provides support for instructors 


2.75 


1.50 


4.75 


My organization communicates well internally about 
learning 


2.50 


4.75 


My organization promotes communication between 
learners and instructors 


3.25 


My organization fosters trust within the organization 


3.75 


My organization uses course data to evaluate classes 


My organization collects user data to understand 
instructor needs 


3.25 


2.25 


My organization collects user data to understand 
student needs 


2.25 


My organization implements in-person (face to face) 
instruction 


5.00 


Appendix E: Survey 


The survey below is a print version of the online survey. Formatting of this document is different 
from what participants received in the live survey. The online survey can be viewed at 


https://asu.co1 .qualtrics.com/jfe/form/SV_cv TwzDbSSWRs TIP 


SoLaR elearning survey 


Please answer the survey questions to the best of your knowledge. If you do not feel comfortable 
answering a question, or do not feel that you know the answer, you may skip any question. 


Your answers to these questions will be anonymous unless you choose to share your name and 
contact information at the end of the survey. If you provide your name and email address, you 
may be contacted by the research team with additional questions or an invitation to participate in 
an interview. 


What is your age? 


What is your gender? 


What is your race/ethnicity? 


What is the highest degree or level of school that you have completed? 


Less than a high school diploma 


High school degree or equivalent (e.g., GED) 


Some college, no degree 


Associate degree (e.g., AA, AS) 


Bachelor's degree (e.g., BA, BS) 


Master's degree (e.g., MA, MS, MEd) 


Professional degree (e.g., MD, DDS, DVM) 


Doctorate (e.g., PhD, EdD) 


If you have completed a degree, what field is your highest degree in? 


What is your current position within the organization? 


Leadership (e.g., Chief Officer of Learning Organization) 


Instructional Design Center Administration 


Training Supervisor 


Instructional Designer 


Instructor 


Student 


Other: 


How many years and months have you held your current position? 


Years 


Months 


How many years/months have you been in the organization (total time)? 


Years 


Months 


How would describe your organization overall? 


Public (e.g., military or government) 


Private (e.g., company or industry) 


Academic (e.g., university, college, or school) 


How would you describe your learning organization? 


K-12 School District 


Trade School 


College or University 


Postsecondary Accreditation Agency 


Licensing or Credentialing Body 


Corporate Human Resources Program 


Military Manpower, Personnel, Training, or Education System 


Industry Association 


International Organizations or NGO 


Other 


Please briefly describe your learning organization. 


332 


Please read the following statements and think about how well they describe your learning 
organization. Please rate how much you agree or disagree with each statement. 


Strongly 
Agree 


My organization 
understands how 
people learn 


My organization values 
learning 


My organization 
provides opportunities 
for learning 


My organization 
provides support for 
learners 


My organization 
provides support for 
instructors 


My organization uses 
technology beyond the 
basic platform 


My organization 
communicates well 
about learning 


My organization 
promotes 
communication between 
learners and instructors 


My organization fosters 
trust within the 
organization 


My organization uses 
course data to evaluate 
classes 


My organization collects 
user data to understand 
instructor needs 


My organization collects 
user data to understand 
student needs 


My organization 
implements in-person 
(face to face) instruction 


Agree 


Somewhat 
Agree 


Somewhat 
Disagree 


Disagree 


Strongly 
Disagree 


Organizations often use several technologies for learning. Please rate how often your learning 
organization incorporates the following technologies in the learning environment. 


Sometime 
Always 
Incorpora s 
Incorpora 
ted Incorpora 
ted 
ted 


telecommunic 
ation (e.g., 
Zoom, Skype, 
videoconferen 
ce) 


web-based 
instruction 


computer- 
based 
instruction 
(e.g., 
instructional 
software) 


simulations, 
virtual reality, 
or augmented 
reality 


digital 
educational 
games 


mobile apps 


social media 
platforms 
(e.g., 
Facebook, 
Yellowdig) 


video 
platforms 
(e.g., 
YouTube, 
Vimeo) 


Seldom 
Incorpora 
ted 


Very 
Rarely 
Incorpora 
ted 


Never 
Incorpora 
ted 


interactive 
systems (e.g., 
intelligent 
tutoring 
systems, 
dynamic 
etexts, 
interactive 
CBL systems) 


synchronous 
computer- 
supported 
collaborative 
learning (e.g., 
a chat room) 


asynchronous 
computer- 
supported 
collaborative 
learning (e.g., 
a discussion 
forum) 


In your personal opinion, please rate the importance of the following technologies for 


successful e-learning. 


Extremely Humerant Somewhat 
Important P Important 


telecommunication 
(e.g., Zoom, Skype, 
videoconference) 


web-based instruction 


computer-based 
instruction (e.g., 
instructional software) 


simulations, virtual 
reality, or augmented 
reality 


digital educational 
games 


mobile apps 


social media platforms 
(e.g., Facebook, 
Yellowdig) 


video platforms (e.g., 
YouTube, Vimeo) 


interactive systems 
(e.g., intelligent tutoring 
systems, dynamic 
etexts, interactive CBL 
systems) 


synchronous computer- 
supported Collaborative 
learning (e.g., a chat 
room) 


asynchronous computer- 

supported collaborative 
learning (e.g., a 
discussion forum) 


Somewhat 
Not 
Important 


Not 
Important 


Extremely 
Not 
Important 


Think about the features of the technologies that your organization uses for learning. Across 
various technologies, please rate how often the following features play a role. 


A 
A lot moderate A little Rarely Not at all 
amount 


A great 
deal 


adaptivity (e.g., 
personalized 
feedback and 
responding) 


artificial intelligence 
(e.g., algorithms to 
analyze user data) 


tutorials (e.g., 
prepare learners to 
use the technology) 


personalized 
learning (e.g., 
customization, 
interactive learning 
environment, 
flexible scheduling 
and pacing, and 
authentic 
assessment) 


dashboards (e.g., 
data visualization of 
student 
performance) 


video (recordings or 
streaming) 


usability evaluations 


In your personal opinion, please rate the importance of the following features for enhancing e- 
learning. 


Somewhat Extremely 
Not 
al Important Net 
Important P Important 


Somewhat 
Important 


Extremely 
Important Important 
adaptivity 
(e.g., 
personalized 
feedback and 
responding) 


artificial 
intelligence 
(e.g., 
algorithms to 
analyze user 
data) 


tutorials (e.g., 
prepare 
learners to 
use the 
technology) 


personalized 
learning (e.g., 
customization, 
interactive 
learning 
environment, 
flexible 
scheduling 
and pacing, 
and authentic 
assessment) 


dashboards 
(e.g., data 
visualization 
of student 
performance) 


video 
(recordings 
or streaming) 


usability 
evaluations 


Organizations often use several instructional methods for learning. Please rate how often your 
learning organization incorporates the following types of instruction in the learning 
environment. 


Very Somewhat Very 
Often Hen Often Rarery Rarely 


in-person 
instruction 


asynchronous 
online 
learning 


synchronous 
online 
learning 


flipped 
classrooms 


MOOCs 
(massive 
open online 
courses) 


mobile 
learning 
(technology 
supported 
out of 
classroom 
learning) 


In your personal opinion, please rate the importance of the following types of instruction for 
successful e-learning. 


Extremely Somewhat Somewhat Not Bateticly 
Important Not Not 
Important 


Faporant Important eee Important 


in-person 
instruction 


asynchronous 
online 
learning 


synchronous 
online 
learning 


flipped 
classrooms 


MOOCs 
(massive 
open online 
courses) 


mobile 


learning 
(technology 
supported 
out of 
classroom 
learning) 


Please share an example of when an e-learning or instructional technology was used effectively. 
Briefly tell us this success story. 


Please share an example of when an e-learning or instructional technology was used 
ineffectively. Briefly tell us about this failure. 


To what extent are the following principles used to support effective e-learning in your 
organization? 


Very 


eave Frequently 


memorization 


motivation 


critical thinking 


problem-solving 


feedback 


self-evaluation 


goal setting 


learning styles 


brain-based learning 


personalized learning 


competency-based 
learning (e.g. 
demonstrated mastery 
of skills that make up 
a domain area) 


social learning 


assessment and 
testing 


team training 


collaborative learning 


Occasionally Rarely 


Very 
Rarely 


Never 


observational learning 
(e.g., modeling, how 
to videos) 


establishing trust in 
the content 


student support 
services (e.g., 
advising, counseling, 
libraries) 


In your personal opinion, please rate the importance of the following principles for e-learning? 


Extremely 


important uportal 


memorization 


motivation 


critical thinking 


problem-solving 


feedback 


self-evaluation 


goal setting 


learning styles 


brain-based learning 


personalized learning 


competency-based 
learning 


social learning 


assessment and testing 


team training 


collaborative learning 


Somewhat 
Important 


Somewhat 
Not 
Important 


Not 
Important 


Extremely 
Not 
Important 


observational learning 
(e.g., modeling, how to 
videos) 


establishing trust in the 
content 


student support services 
(e.g., advising, 
counseling, libraries) 


What should your organization do to better support e-learning? How should your organization 
promote effective learning via technology? Please briefly share one or two recommendations. 


If you feel comfortable doing so, please share your name and contact information (email 
address) below. Providing this information is entirely optional and is confidential. Your 
information will not be shared publicly or with your organization. 


We may contact you with follow-up questions or to conduct a short interview. 
Name (4) 


Email Address (5) 


